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Executive Summary 



E vidence is mounting that where and how youth spend their time outside 
of normal school hours has important implications for their develop- 
ment. On the negative side, estimates suggest that more than 7 million 
children in the United States are without adult supervision for some period of 
time after school. This unsupervised time puts youth at risk for such negative 
outcomes as academic and behavioral problems, drug use and other types of 
risky behavior (Weisman & Gottfredson, 2001). On the positive side, young 
people benefit when they spend time engaged in structured pursuits that offer 
opportunities for positive interactions with adults and peers, encourage them 
to contribute and take initiative, and contain challenging and engaging tasks 
that help them develop and apply new skills and personal talents (American 
Youth Policy Forum, 2006; Carnegie Corporation, 1992; Larson & Verma, 1999; 
National Research Council & Institute of Medicine, 2002). 

As a result, there has been increasing interest in after-school programs 
(ASPs) that can provide youth with a safe and supportive adult-supervised 
environment and offer them various growth-enhancing opportunities, including 
activities and experiences that promote academic, personal, social and recre- 
ational development. There is strong public support for after-school programs, 
particularly from working parents who cannot be with their children immedi- 
ately after school. Funding from state, private and federal sources has support- 
ed existing ASPs and created new offerings in many communities. The federal 
government invested K3.6 billion in after-school programs in 2002 
(http://www.financeprojectinfo.org/Publications/FundingGuide2003.pdf). 

What is known about the impact of after-school programs? Previous reviews 
have concentrated on the academic benefits of programs that offer tutoring or 
other forms of academic assistance to youth, and the results have been mixed. 
One review of 35 studies reported that the test scores of low-income, at-risk 
youth improved significantly in both reading and mathematics after they par- 
ticipated in after-school programs (Lauer et al., 2006). Academic outcomes for 
other youth, however, have been inconsistent (Kane, 2003; Scott-Little, 
Hamann & Jurs, 2002; Vandell et al., 2004; Zief, Lauver & Maynard, 2004). As 
a result, authors have stressed the need for careful evaluations of the effective- 
ness of different programs and the factors associated with positive outcomes, 
along with realistic expectations about the academic gains that can be achieved 
(Bodilly & Beckett, 2005; Granger & Kane, 2004; Vandell et al., 2004, 2005). 

However, the personal and social benefits of after-school programs have been 
somewhat overlooked, at least in terms of formal evaluation. Many acknowl- 
edge that after-school programs can improve young people’s personal and social 
development, and findings from some individual studies have been positive 
(e.g.. Harvard Family Research Project, 2003). But no review has been done to 
evaluate systematically the impact of after-school programs that attempt to 
enhance youths’ personal and social skills, identify the nature and magnitude 
of the outcomes of such programs, and describe the features that characterize 
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effective programs. These are the goals of the current review. 

All the programs in the current review were selected because their overall 
mission included promoting young people’s personal and social development. 
Many programs offer a mix of activities, but the current review concentrates on 
those aspects of each program that are devoted to developing youths’ personal 
and social s ki lls. 

There is extensive evidence from a wide range of promotion, prevention and 
treatment interventions that youth can be taught personal and social skills 
(Collaborative for Academic, Social, and Emotional Learning, 2003; 

Commission on Positive Youth Development, 2005; L’Abate & Milan, 1985; 
Greenberg et al., 2003). Moreover, theory and research about skills training of 
children and adolescents indicate that learning is more likely to occur when evi- 
dence-based training approaches are used (Collaborative for Academic, Social, 
and Emotional Learning, 2003; Durlak, 1997, 2003; Elias et al., 1997; National 
Research Council & Institute of Medicine, 2002; Payton et al. 2000; Weissberg & 
Greenberg, 1998). Effective approaches to skills development are sequential, 
active, focused and explicit. Knowing this, we hypothesized that programs that 
used all four approaches to promote youths’ personal and social skills would be 
more successful than those that did not, and we developed a method to capture 
the application of these evidence-based approaches. (The rationale and coding 
methodology for these variables are described in the full report.) 

We expected that youth would benefit in multiple ways from effective pro- 
gramming, so we examined outcomes in three general areas: feelings and atti- 
tudes, indicators of behavioral adjustment, and school performance. Our objec- 
tive was to answer two research questions: 

1. What types of outcomes can we expect from after-school programs that 

attempt to foster young people’s personal and social s ki lls? 

2. Gan we identify program characteristics that are associated with better 

results? 

Method 

We only considered after-school programs that attempted to promote personal 
and social skills. The personal and social skills targeted in these programs 
could include one or more skills in such areas as problem-solving, conflict reso- 
lution, self-control, leadership, responsible decision-making, and enhancement 
of self-efficacy and self-esteem. We defined after-school programs as interven- 
tions that were offered to children between the ages of 5 and 18, operated dur- 
ing at least part of the school year (i.e., September to June) and occurred out- 
side of normal school hours, which are typically 8 a.m. to 2:30 p.m., Monday 
through Friday. To be included, reports had to have a control group, present 
sufficient information for analysis and appear by Dec. 31, 2005. 

A careful and systematic search for published and unpublished studies net- 
ted a set of reports that provided information on 73 programs. We conducted a 
meta-analysis to evaluate the magnitude of effects obtained from each program. 
This summary focuses on the major findings. The technical aspects of the 
analyses are contained in the full report. 
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Results 

The two most important findings were: 

1. Youth who participate in after-school programs improve significantly m 
three major areas: feelings and attitudes, indicators of behavioral adjust- 
ment, and school performance. More specifically, after-school programs suc- 
ceeded in improving youths’ feelings of self-confidence and self-esteem, 
school bonding (positive feelings and attitudes toward school), positive social 
behaviors, school grades and achievement test scores. They also reduced 
problem behaviors (e.g., aggression, noncompliance and conduct problems) 
and drug use. In sum, after-school programs produced multiple benefits that 
pertain to youths’ personal, social and academic life. 

2. It was possible to identify effective programs: Programs that used evidence- 
based s kill training approaches were consistently successful in producing 
multiple benefits for youth, while those that did not use such procedures 
were not successful in any outcome area. 

Evidence-Based Training Approaches: 

Drawing on theory and research about skills training, we applied two criteria 
related to the training process and two criteria related to program content to 
identify programs that used evidence-based training approaches to promote 
personal and social skills. The two criteria related to process were the presence 
of a sequenced set of activities to achieve skill objectives (sequenced), and the 
use of active forms of learning (active). The two criteria related to content were 
the presence of at least one program component focused on developing person- 
al or social skills (focus), and the targeting of specific personal or social skills 
(explicit). 

Thirty-nine programs met all four of the above criteria, while 27 programs 
did not. When we compared the outcomes from the two sets of programs, a 
clear pattern emerged: The former programs yielded significant positive results 
on all seven of the outcome categories mentioned above (improved feelings of 
self-confidence and self-esteem, school bonding, positive social behaviors, 
school grades and achievement test scores, together with reduced problem 
behaviors and drug use), while the latter did not produce positive results for 
any category. When it comes to enhancing personal and social skills, effective 
programs are SAFE — sequenced, active, focused and explicit. 

Discussion 

There are at least three reasons why our findings should be deemed credible. 

1. We searched the literature carefully and systematically for relevant reports, 
and assembled a representative and unbiased sample of published and 
unpublished evaluations. (Indeed, many of the reports were scrutinized for 
the first time for our review.) We evaluated a large number of after-school 
programs (n=73). Sixty percent of the evaluated reports appeared after 2000. 
As a result, this review presents an up-to-date perspective on a rapidly grow- 
ing body of research literature. 

2. We only considered reports that included control groups. 
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3. To substantiate the findings regarding the eharaeteristies of effeetive pro- 
grams, in our analyses we eontrolled for the possible influenee of several 
methodologieal features found in the reports. 

Current data offer elear empirieal support for the eonelusion that well-run 
ASPs ean produee a variety of positive benefits for partieipating youth. More 
speeifieally, there is signifieant improvement in youths’ feelings and attitudes 
(i.e., their self-pereeptions and bonding to sehool), their behavioral adjustment 
(i.e., inereases in positive soeial behaviors and deereases in problem behaviors 
and drug use), and in their sehool grades and level of aeademie aehievement. 

We eonfirmed that effeetive programs employed skill-development aetivities 
that were sequential, aetive, foeused and explieit. It is important to stress that 
only those programs that followed these four evidenee-based training approaeh- 
es in their program eomponents devoted to skill development produeed signifi- 
eant ehanges in any outeomes. In other words, it is the eombination of both 
training proeess (i.e., sequential and aetive) and program eontent (i.e., foeused 
and explieit) that leads to positive results. 

On the basis of these results, we strongly reeommend that after-sehool pro- 
grams seeking to promote personal and soeial skills use the evidenee-based 
approaehes deseribed in this report. (Others have mentioned the importanee 
one or more of these features in after-sehool programs as well: see Larson & 
Verma, 1999; Miller, 2003; National Researeh Gouneil & Institute of Medieine, 
2002). Not only ean partieipants benefit in multiple ways if these eomponents 
are ineluded, but sueeess is unlikely if they are missing. To improve youths’ 
personal and soeial skills, programs must devote suffieient time to skill 
enhaneement, be explieit about what they wish to aehieve, use aetivities that 
are eoordinated and sequeneed to aehieve their purpose, and require aetive 
involvement on the part of partieipants. 
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Abstract 



A meta-analysis of after-school programs (ASPs) that seek to enhance the 
personal and social development of children and adolescents indicated 
that youth improved in three general areas: feelings and attitudes, indi- 
cators of behavioral adjustment, and school performance. More specifically, sig- 
nificant increases occurred in youths’ self-perceptions and bonding to school, 
their positive social behaviors, and in their school grades and level of academic 
achievement. At the same time, significant reductions occurred in problem 
behaviors and drug use. Substantial differences emerged between programs 
that used evidence-based approaches for skill training and those that did not. 
The former programs consistently produced significant improvements among 
participants in all of the above outcome areas (mean effect sizes ranged from 
0.24 to 0.35), whereas the latter programs did not produce significant results in 
any outcome category. Our findings have two important implications for future 
research, practice and policy. The first is that ASPs should contain components 
to foster the personal and social skills of youth, because participants can bene- 
fit in multiple ways if these components are offered. The second is that such 
components are effective only if they use evidence-based approaches. When it 
comes to enhancing personal and social skills, successful programs are SAFE — 
sequenced, active, focused and explicit. 
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E vidence is mounting that where and how youth spend their time outside 
of normal school hours has important implications for their develop- 
ment. On the negative side, estimates suggest that more than 7 million 
children in the United States are without adult supervision for some period of 
time after school. This unsupervised time puts youth at risk for such negative 
outcomes as academic and behavioral problems, drug use and other types of 
risky behavior (Weisman & Gottfredson, 2001). On the positive side, young 
people benefit when they spend time engaged in structured pursuits that offer 
opportunities for positive interactions with adults and peers, encourage them 
to contribute and take initiative, and contain challenging and engaging tasks 
that help them develop and apply new skills and personal talents (American 
Youth Policy Forum, 2006; Carnegie Corporation, 1992; Eccles & Templeton, 
2002; Larson & Verma, 1999; National Research Council & Institute of 
Medicine, 2002). 

As a result, there has been increasing interest in the value of formal after- 
school programs (ASPs) that can provide youth with a safe and supportive 
environment that is supervised by adults and offers various growth-enhancing 
opportunities, including activities and experiences that promote young people’s 
academic, personal, social, recreational and cultural development. There is 
strong public support for ASPs, particularly from working parents who cannot 
be with their children immediately after school. Funding from state, private 
and federal sources has supported existing ASPs and created new offerings in 
many communities. For example, the federal government invested 03.6 billion 
in after-school programs in 2002 (see http://www.financeprojectinfo.org/ 
Publications/FundingGuide2003 .pdf) . 

What is known about the impact of ASPs? Previous reviews have concentrated 
on the academic benefits of programs that offer tutoring or other forms of aca- 
demic assistance to youth, but the results have been mixed. One meta-analysis of 
35 studies reported that the test scores of low-income, at-risk youth improved 
significantly in both reading and mathematics after they participated in ASPs 
(Lauer et al., 2006). Academic outcomes for other youth, however, have been 
inconsistent (Kane, 2003; Scott-Little, Hamann & Jurs, 2002; Vandell et al., 2004; 
Zief, Lauver & Maynard, 2004). As a result, some authors have stressed the need 
for careful evaluations of the effectiveness of different programs and the factors 
associated with positive outcomes, along with realistic expectations about the 
academic gains that can be achieved through ASPs (Bodilly & Beckett, 2005; 
Granger & Kane, 2004; Kane, 2003; Vandell et al., 2004, 2005). 

However, the potential personal and social benefits of ASPs have been some- 
what overlooked. Several authors have stressed that ASPs can improve young 
people’s personal and social development, and findings from several studies 
have been positive (e.g.. Harvard Family Research Project, 2003). But no 
review has been done to evaluate systematically the impact of ASPs that 
attempt to enhance youths’ personal and social skills, identify the nature and 
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magnitude of the outeomes of sueh programs, and deseribe the features that 
eharaeterize effeetive programs. These are the goals of the eurrent review. 

There are a number of variations among ASPs, ineluding their goals, where 
they are housed, struetural and organizational features sueh as staffing pat- 
terns, staff-student ratios, budgets and operating hours, and the extent to 
whieh aetive parent involvement and eommunity eontaets and support are 
sought and obtained. All the programs ineluded in the eurrent review were 
seleeted beeause their overall mission ineluded promoting young people’s per- 
sonal and soeial development. Although some ASPs offer a mix of aetivities that 
inelude aeademie, soeial, eultural and reereational pursuits, the eurrent review 
eoneentrates on those aspeets of eaeh program that are devoted to developing 
youths’ personal and soeial skills. 

There is extensive evidenee from a wide range of promotion, prevention and 
treatment interventions that youth ean be taught personal and soeial s ki lls 
(Beelman, Pfingsten & Losel, 1994; Gartledge & Milburn, 1980; Collaborative 
for Aeademie, Soeial, and Emotional Learning, 2003; Commission on Positive 
Youth Development, 2005; L’Abate & Milan, 1985; Losel & Beelman, 2003; 
Greenberg et al., 2003). These s ki lls eover sueh areas as self-awareness and 
self-management (e.g., self-eontrol and self-effieaey), soeial awareness and 
soeial relationships (e.g., problem-solving, eonfliet resolution and leadership 
skills) and responsible deeision-making. 

Theory and researeh about skills training of ehildren and adoleseents indi- 
eate that learning is more likely to oeeur when evidenee-based training 
approaehes are used. Several sourees diseuss how these approaehes ean be 
applied in interventions for youth (Collaborative for Aeademie, Soeial and 
Emotional Learning, 2003; Durlak, 1997, 2003; Elias et al., 1997; National 
Researeh Gouneil & Institute of Medieine, 2002; Payton et al., 2000; Weissberg 
& Greenberg, 1998). Therefore, we hypothesized that ASPs that used these evi- 
denee-based approaehes to promote youths’ personal and soeial skills would be 
more sueeessful than those that did not, and we developed eodes to eapture 
the applieation of sueh approaehes. More speeifieally, we expeeted that 
enhaneed skill development would oeeur when relevant programming was 
sequeneed, aetive, foeused and explieit (SAFE). The rationale and eoding for 
these variables are deseribed in the Method seetion. 

We expeeted that youth would benefit in multiple ways from effeetive pro- 
gramming, so we examined outeomes in three areas: feelings and attitudes, indi- 
eators of behavioral adjustment, and sehool performanee. Skill-oriented sehool- 
based interventions have obtained positive outeomes in these three areas 
(Durlak & Weissberg, 2005; Weissberg, Durlak, Dymnieki & Taylor, 2006), and 
we expeeted a similar pattern of findings would emerge for sueeessful ASPs. 

Our objeetive was to answer three basie researeh questions: 

1. What types of outeomes ean we expeet from ASPs that attempt to foster the 
personal and soeial skills of program partieipants? 

2. How mueh ehange oeeur s in different areas? 

3. Gan we identify program eharaeteristies that are assoeiated with better 
results? 
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W e defined an ASP as one or more aetivities that: (1) operated during 
at least part of the sehool year; (2) oeeurred outside of normal 
sehool hours; and (3) were supervised or in some way monitored by 
adults. In addition to meeting this definition, the ASP had to have as one of its 
goals the development of one or more personal or soeial skills in young people 
between the ages of 5 and 18. These eould inelude skills in sueh areas as prob- 
lem-solving, eonfliet resolution, self-eontrol, leadership, responsible deeision- 
making, and enhaneement of self-effieaey and self-esteem. To be ineluded, 
reports also had to have a eontrol group, present suffieient information so that 
effeet sizes eould be ealeulated, and appear by Dee. 31, 2005. Although it was 
not a formal eriterion, all the reports deseribed programs eondueted in the 
United States. 

We exeluded reports based on some methodologieal and eon tent grounds. In 
the former ease, single group pre-post studies and studies not amenable to 
meta-analysis due to insuffieient reporting of data were exeluded. In the latter 
ease, we exeluded ASPs that foeused on aeademie performanee or sehool atten- 
danee and only reported sueh outeomes, adventure edueation and Outward 
Bound programs, extraeurrieular sehool aetivities and summer eamps. 

“Out-of-sehool time” is a term that is being used inereasingly to refer to all 
aetivities oeeurring outside typieal sehool hours, ineluding extraeurrieular 
sehool aetivities, aeademie and reereational programs eondueted during the 
summer, and edueational and soeial events offered by loeal libraries, museums, 
parks and faith-based institutions. We did not inelude these types of aetivities 
in our review, but information on the benefits that ean result from sueh aetivi- 
ties is available (Bodilly & Beekett, 2005; Cason & Gillis, 1994; Cooper, 
Charlton, Valentine & Muhlenbruek, 2000; Feldman & Matjasko, 2005; Harvard 
Family Researeh Projeet, 2003). 

Locating Relevant Stndies 

The primary goal of the seareh proeess was to seeure a nonbiased, representa- 
tive sample of studies obtained through a systematie seareh for published and 
unpublished reports. We used five proeedures to loeate reports: 

1. Computer searehes of multiple databases (ERIC, Psyelnfo, Medline and 
Dissertation Abstracts) 

2. Manual searehes of the eontents of several journals that published after- 
sehool outeome studies 

3. Inspeetion of the referenee lists of previous ASP reviews and the referenees 
of eaeh ineluded report 

4. Searehes of web sites hosted by organizations involved in after-sehool aetivi- 
ties 

5. Requests made to researehers, praetitioners and poliey advoeates through 
eontaets developed by the funding ageney (William T. Grant Foundation) 

The literature seareh ended Dee. 31, 2005. We loeated and seeured many 
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unpublished reports. Although no review can be absolutely exhaustive, we 
believe that the study sample is a representative group of current program eval- 
uations. 



Study Sample 

In this report, we evaluate the results from 73 ASPs that were described in 49 
reports. Several reports presented data on more than one ASP, each with its 
own control group, and these interventions were treated as separate programs. 
(See Appendix B for details.) Of the 73 programs, 66 assessed outcomes at post 
(i.e., immediately following the intervention); seven also collected some follow- 
up information; and seven contained only follow-up data. We focus first on the 
post-data from 66 programs and then present the follow-up data. Post-effects 
were based on the endpoint of the youths’ program participation. That is, if two 
reports were available on the same program and one contained results after 
one year of participation while the second offered information after two years 
of participation, only the latter data were evaluated. The bibliography of all 
included studies is presented in Appendix A. 

Standardized Mean Differenees As Outeomes 

The index of effect was a standardized mean difference (ES) that was calculat- 
ed whenever possible by subtracting the mean of the control group at post 
from the mean of the after-school group at post and dividing by the pooled 
standard deviation of the two groups. If means and standard deviations were 
not available, effects were estimated using procedures described by Lipsey and 
Wilson (2001). When results were reported as nonsignificant and no other 
information was available, the effect size for that outcome measure was conser- 
vatively set at zero. Each effect was corrected for small sample bias and effects 
were weighted by the inverse of their variance prior to any analysis (Hedges & 
Olkins, 1985). Higher effects are desired and reflect a stronger positive impact 
on the after-school group compared to controls. Whenever possible, we adjust- 
ed for any pre-intervention differences between groups on each outcome meas- 
ure by first calculating a pre-ES and then subtracting this pre-ES from the 
obtained post-ES. This strategy has been used in other meta-analyses (Derzon, 
2006; Wilson, Gottfredson & Najaka, 2001) and is helpful when dealing with 
many quasi-experimental designs. 

Our consistent strategy in treating effect sizes was to calculate one effect 
size per study for each analysis. In other words, for the first analysis of the 
overall effects from all 66 programs at post, we averaged all the effect sizes 
within each study so that each study yielded only one effect. For the later 
analyses by outcome category, if there were multiple measures from a program 
for the same outcome category, we averaged them so that each study con- 
tributed only one effect size for that outcome. For example, if data for stan- 
dardized test scores were available for multiple areas such as reading and 
math, the data were averaged to produce a single effect reflecting academic 
achievement. 

The analyses used a random effects model. This procedure adds an error 
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term to the ealeulation of effeets in eonsideration of the unique features of 
eaeh program evaluation. A random effeets model permits interpretation of 
findings to be generalized to all types of ASPs. 

In testing the statistieal signifieanee of obtained effeets, we used a two-tailed 
.05 probability level throughout. Mean effeets for different study groupings are 
reported along with ±.05 eonfidenee intervals (GI). Means whose eonfidenee 
intervals do not inelude zero differ signifieantly from zero at the .05 level. In 
the Diseussion seetion we eompare eurrent findings to those obtained in meta- 
analyses of other interventions and diseuss how effeets ean be understood in 
praetieal terms. 

Coding 

We developed a eoding system to eapture basie study features and methodolog- 
ieal aspeets of the program evaluation, as well as eharaeteristies of the ASP, 
partieipants and outeomes. The eoding of most of the variables is straightfor- 
ward and explained in the manual, whieh is available from the authors upon 
request. The eoding for a few variables is deseribed below. 

Methodological Features 

Three primary methodologieal features were eoded: use of a randomized 
design, problems with attrition and the reliability of the outeome measure. 

“Problems with attrition” refers to loss of data from the program and eontrol 
groups, and was eoded when attrition was >10%. In addition, when higher rates 
of attrition oeeurred, problems were eoded as present if the author failed to: (a) 
eheek for differential attrition aeross program and eontrol eonditions; (b) 
examine the equivalenee of eontinuers and dropouts on key variables; or (e) 
eonduet any analyses in the presenee of differential attrition to examine if this 
event affeeted outeomes. 

The reliability of an outeome measure was eonsidered aeeeptable if its alpha 
eoeffieient was >.70, or an assessment of inter-judge agreement for eoded or 
rated variables was >.70 (for kappa, >.60). 

Outcome Categories 

We analyzed data for outeomes grouped into eight eategories. Two of these 
assessed feelings and attitudes (ehild self-pereeptions and bonding to sehool); 
three were indieators of behavioral adjustment (positive soeial behaviors, prob- 
lem behaviors and drug use); and three assessed aspeets of sehool performanee 
(performanee on aehievement tests, grades and sehool attendanee). 

Self-pereeptions ineluded measures of self-esteem, self-eoneept, self-effieaey 
and, in a few eases (four studies), raeial/eultural identity or pride. Sehool bond- 
ing assessed positive feelings and attitudes toward sehool or teaehers (e.g., lik- 
ing sehool, or reports that the sehool/elassroom environment or teaehers are 
supportive). Positive soeial behaviors measured positive interaetions with oth- 
ers. These are behavioral outeomes that assess sueh things as effeetive expres- 
sion of feelings, positive interaetions with others, eooperation, leadership, 
assertiveness in soeial eontexts or appropriate responses to peer pressure or 
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interpersonal eonfliet. Problem behaviors assessed diffieulties that youth 
demonstrated in eon trolling their behavior adequately in soeial situations. This 
eategory ineluded noneomplianee, aggression, delinquent aets, diseiplinary 
referrals, rebelliousness and other types of eonduet problems. Drug use, whieh 
individual youths self-reported, usually involved the eonsumption of aleohol, 
marijuana or tobaeeo; less frequently reported substanees ineluded inhalants 
and illieit drugs sueh as eoeaine and amphetamines. The aehievement tests 
eategory refleeted performanee on standardized sehool aehievement tests, pri- 
marily those that assess reading and mathematies skills. Sehool grades were 
either drawn from sehool reeords or reported by youth, and refleeted either 
performanee in speeifie subjeets sueh as reading, mathematies or soeial stud- 
ies, or overall grade point average. Sehool attendanee assessed the frequeney 
with whieh students attended sehool. 

Evidence-Based Training Approaches 

Drawing on theory and researeh about skills training, we established two erite- 
ria related to the training proeess and two eriteria related to program eontent. 

We eoded the training proeess aeeording to whether or not it was sequeneed 
(i.e., did the program use a sequeneed set of aetivities to aehieve the objeetives 
related to skill development?) and aetive (i.e., did the program use aetive forms 
of learning to help youth learn new skills?). 

New skills eannot be aequired instantaneously. It takes time and effort to 
develop any new behaviors, and relatively eomplieated skills often must be bro- 
ken down into smaller steps and mastered sequentially. Therefore, a eoordinat- 
ed sequenee of aetivities is required, one that links the learning steps and pro- 
vides youth with opportunities to eonneet them. These sequeneed aetivities are 
usually laid out in lesson plans or program manuals, partieularly if programs 
use or adapt established eurrieula. 

Different young people have different learning styles, and some ean learn 
through a variety of teehniques. However, evidenee from many edueational and 
psyehosoeial interventions indieates that the most effeetive and effieient teaeh- 
ing strategies for many youth emphasize aetive forms of learning. Young people 
learn best by doing. 

Aetive forms of learning require youth to aet on the material. That is, after 
they reeeive some basie instruetion, they should have the opportunity to prae- 
tiee new behaviors and reeeive feedbaek on their performanee. This is aeeom- 
plished through role playing and other types of behavioral rehearsal strategies. 
A eyele of praetiee and feedbaek eontinues until mastery is aehieved. Hands-on 
forms of learning are mueh preferred over exelusively didaetie instruetion, 
whieh rarely translates into behavioral ehange (Durlak, 1997). 

Program eontent was eoded for foeus (i.e., did the program have at least one 
eomponent devoted to developing personal or soeial skills?) and explieitness 
(i.e., did the program target speeifie personal or soeial skills?). 

Suffieient time and attention must be devoted to any task for learning to 
oeeur. Therefore, programs should designate time that is primarily for s ki ll 
development. Furthermore, elear and speeifie learning objeetives are preferable 
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to general ones. Youth need to know what they are expeeted to learn. 
Therefore, programs should not foeus on personal and soeial development in 
general, but identify explieitly what skills in these areas youth are expeeted to 
learn (e.g., self-eontrol, problem-solving skills, resistanee skills, and so on). 

We eoded eaeh of the above four program features as either present or 
absent. Beeause researeh indieates that both eontent and proeess are impor- 
tant in training, programs that met all four eriteria were eonsidered to be using 
evidenee-based training approaehes, while those not meeting all four eriteria 
were not. 

Reliability of Coding 

For all variables but one, we estimated reliability by randomly seleeting 25% of 
the studies and ending them independently. (Coding was done by trained grad- 
uate student assistants and the first author.) For the eriteria regarding use of 
evidenee-based training proeedures, all the studies were independently rated 
by researeh assistants and their data were then eompared with the first 
author’s. Kappa eoeffieients eorreeted for ehange agreement were aeeeptable 
aeross all eodes (.70 to .95, average=.85) and the disagreements in ending were 
resolved through diseussion. The produet moment eorrelations for eontinuous 
items ineluding the ealeulation of effeets were all above 0.95. 
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Results 



Table 1: Descriptive 
Characteristics 
OF Reviewed Studies 



N % 



T able 1 summarizes several features of the 66 studies with post-data. 

Additional information on all 73 interventions is ineluded in Appendix 
B, whieh notes the author and date of eaeh report; the name of the pro- 
gram when relevant (e.g., 21st Century Learning Centers); the study level 
effeet size for eaeh program; and information about program loeation (sehool 
eampus or non-sehool-based), duration, and the presenee of aeademie eompo- 
nents, aetive parent involvement and use of evidenee-based training approaeh- 
es. Appendix B also ineludes information on how we handled reports that eon- 
tained multiple samples. 

Returning to Table 1, 60% of the 66 post-studies had appeared after 2000, 
and the majority were unpublished teehnieal reports or dissertations (n=44, or 
67%). In terms of partieipants, only a small number served youth who were pri- 
marily or only of high sehool age (n=5, or 8%), whereas nearly half served ele- 
mentary students (i.e., those in grades 1 to 6; n=31, or 47%) and more than a 
third served students in junior high (grades 7-9; n=24, or 36%). 

Every meta-analysis finds that information on some variables is missing. With 
this review, we eneountered the problem most frequently with information 
about the ethnieity and soeioeeonomie status of the partieipants. For example, 
46 studies did not speeify the ethnieity of the partieipants, and the remaining 
20 reported this information in different ways. Among the latter studies, partiei- 
pating youth were predominantly Afriean Ameriean in 10 studies. Latino in six 
studies, Asian or Paeifie Islander in three studies and Ameriean Indian in one 
study (i.e., the speeified raee/ethnieity eonstituted >90% of the sample). 

There was no information on the soeioeeonomie status of the partieipants’ 
families in more than half of the reports (n=36, or 55%). Based on the way 
information was reported in the remaining studies, 17 studies (26%) primarily 
served a low-ineome group and 13 studies (20%) served youth from both low- 
and middle-ineome levels. 

Methodological Features 

Table 1 also presents information about three methodologieal features, namely, 
the pereentage of studies that employed randomized designs (26%), the per- 
eentage of studies that had no problems with attrition (85%), and the pereent- 
age of all outeome measures for whieh reliability was reported and was aeeept- 
able (69%). (The ending manual explains how these variables were ended.) 

Program Characteristics 

As indicated in Table 1, more programs were conducted in community settings 
than on school grounds (56% versus 41%) and many had an academic compo- 
nent (i.e., tutoring or homework assistance, 41%). Youth had participated for 
less than one year when the program evaluation was conducted in the majority 
of the reports (70%). Active parent involvement included such services as par- 
enting or child development workshops, ESL classes or support groups. 



PUBLICATIDN FEATURES 

Date of report 



1983-1990 


3 


5 


1991-2000 


23 


35 


2001-2005 


40 


60 


Source of report 


Published article 


22 


33 


Unpublished dissertation 


7 


11 


Unpublished technical 


report, program evaluation 


37 


56 


Methodological Features 




Randomization 


Randomized 


17 


26 


Non-randomized 


49 


73 


Reliability of outcome measures 




Acceptable reliability 


283 


69 


Unknown/unacceptable 


127 


31 


Problems with attrition 


Yes 


10 


15 


No 


56 


85 


Participant Characteristics 




Mean educational level of participants 


Elementary school 


31 


47 


Junior high school 


24 


36 


High school 


5 


8 


Missing 


6 


9 


Presenting problems 
None (universal 


intervention) 


59 


89 


Some presenting problems 


7 


11 


Predominant ethnicity of participants 


African-American 


10 


15 


Latino 


6 


9.1 


Asian/Pacific Islander 


3 


4.5 


American Indian 


1 


1.5 


Did not report data 


46 


70 


Socioeconomic status 


Low-income 


17 


25.7 


Mixed income 


13 


19.7 


Did not report data 


36 


54.5 


Prdgram Features 
D uration 


Less than one year 


46 


70 


One to two years 


10 


15 


More than two years 


10 


15 


Setting 


On school grounds 


27 


41 


In community 


37 


56 


Unknown/not reported 


2 


3 


Academic components 
Tutoring/homework 


assistance 


28 


42 


No academic component 


38 


58 


Use of evidence-based 
training procedures 


Yes 


39 


59 


No 


27 


41 


Active parent involvement 


Yes 


24 


36 


No 


42 


64 



Note: Percentages do not always add to 
10096 due to missing data. 
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Table 2: Stem and Leaf 
Distribution of Winsorized 
Study-level Effect Sizes 

Stem & Leaf 
-01 • 65 

00 • -8-5-5-4-3-1-1000+ 

13456777899 

01 • 0113344666789 

02 • 345679 

03 • 0111256899 

04 • 38 

05 • 3479 

06 • 2467 

07 • 46 

08 • 5 

Stem width = 0.10 

Extreme values = 0.90, 0.98, 1.19 

Note: A winsorized effect refers to an effect 
initially at the extreme of the distribution 
that is eventually reset to a less extreme 
value. In the current distribution, this 
referred to three values — 0.90, 0.98 and 
1.19 — that were reset to 0.85. The 
distribution can be read as one effect size 
at +0.85 (before winsorizing), one at 
+ 0.74, one at +0.76, and so on. 



A. Overall Impact 

Our first major finding is that ASPs have an overall positive and statistieally 
signifieant inipaet on partieipating youth. First, we inspected the distribution 
of effects and sample sizes and winsorized values that were three or more stan- 
dard deviations from the mean.^ Table 2 presents a stem-and-left plot of the 
winsorized study level effect sizes. The winsorized study level effects, which 
ranged in value from -0.16 to h- 0.85, had an overall mean of h- 0.22 (GI=0.16 to 
0.29), which was significantly different from zero. However, there was also sig- 
nificant variability in the distribution of effects (p <.001). These results indi- 
cate that there is an overall positive benefit from ASPs, but also suggest the 
need to search for variables that explain the variability in program impact. 



Table 3: Mean Effects for Different Outcomes in Participating Youth 



Outcomes 


Mean Effect Size 


N 


9596 Confidence Interval 


Feelings and Attitudes 


Child self-perceptions 




0.34* 


22 


0.23-0.45 


School bonding 




0.14* 


28 


0.03-0.25 


Indicators of Behavioral Adjustment 
Positive social behaviors 


0.19* 


35 


0.10-0.28 


Problem behaviors 




0.18* 


42 


0.10-0.26 


Drug use 




0.11* 


27 


0.01-0.21 


School Performance 


Achievement tests 




0.16* 


20 


0.05-0.27 


School grades 




0.11* 


25 


0.01-0.22 


School attendance 




0.10 


21 


-0.01-0.20 



* Denotes mean effect is significantly different from zero at tbe .05 level 

Explanation of Outcome Categories 

Feelings and Attitudes 

Self-perceptions: This category included measures of self-esteem, self-concept, self-efficacy and in a few cases (4 
studies) racial/cultural identity or pride. 

School bonding: These measures assess positive feelings and attitudes toward school or teachers (e.g., liking school, or 
reports that the school/classroom environment or teachers are supportive). 

Indicators of Behavioral Adjustment 

Positive social behaviors: These measures assess positive interactions with others. These are behavioral outcomes 
assessing such things as effective expression of feelings, positive interactions with others, cooperation, leadership, 
assertiveness in social contexts, or social skills in general. 

Problem behaviors: These outcomes assessed problems that youth demonstrated in controlling their behavior 
adequately in social situations. This dimension included different types of acting-out behaviors such as noncompliance, 
aggression, delinquent acts, disciplinary referrals, rebelliousness, and other types of conduct problems. 

Drug use: These outcomes are youth self-reports of drug use. Some studies collect data on specific drugs (usually alcohol, 
marijuana, or tobacco); others inquire about overall drug use. If the use of different drugs was assessed on separate measures 
(e.g., use of alcohol, use of marijuana), the effects were averaged to obtain a measure of overall drug use for each study. 

School Performance 

Achievement tests: These outcomes reflect performance on standardized school achievement tests primarily assessing 
reading and mathematics. In the few studies that collected data in multiple areas (e.g., reading and math), the effects 
were averaged to produce a single effect per study. 

School grades: These outcomes were either drawn from school records or reported by youth in areas such as reading, 
mathematics, social studies, or overall grade point average. Once again, grades from multiple subject areas occurring in 
the same study were averaged to produce a single effect for grades. 

School attendance: This outcome assessed the frequency with which students attended school. 



although most programs did not contain these types of aetivities (65%)\ 




^ The programs tltat had brief occasional meetings with parents or telephone or newsletter contact to 
irtfbrm the family about ASP activities were not counted as emphasizing active parent involvement 

^ Winsorizing is a common practice in meta-analysis and is done to accomplish two ends: (1 ) 
to retain as much data as possible for analysis and (2) to reduce the likelihood that extreme 
values will unduly distort the results. 
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B. In Wliat Ways Do Youth Change? 

Our second major finding is that yondi who participate in ASPs improve signifi- 
candy in three major areas: feelings and attitudes, indicators of behavioral adjust- 
ment, and school performance. More specifically, we grouped effect sizes into eight 
major outcome categories. Two of these categories related to feelings and attitudes: 
child self-perceptions and bonding to school; three were indicators of behavioral 
adjustment: positive social behaviors, problem behaviors and drug use; and three 
pertained to school performance: performance on academic achievement tests, 
school grades, and school attendance. Table 3 presents the mean effects obtained 
for each of these eight outcome categories, their confidence intervals, and the num- 
ber of studies containing a measure for each category. The note attached to Table 3 
explains the types of measures that were included in each outcome grouping. 
Appendix G contains a list of the measures drawn from each study that fell into 
each outcome category along with the measure’s effect size. 

Significant mean effects ranged in magnitude from 0.11 (for drug use and school 
grades) to 0.34 for child self-perceptions (i.e., increased self-confidence and self- 
esteem). The mean effect for school attendance (0.10) is the only outcome that 
failed to reach statistical significance. In other words, ASPs have been able to signif- 
icantly increase participants’ positive feelings and attitudes about themselves and 
their school (child self-perceptions and school bonding), and their positive social 
behaviors. In addition, both externalizing behavior and drug use are significandy 
reduced. Finally, there is significant improvement in students’ performance on 
achievement tests and in their school grades. In sum, participation in ASPs leads to 
multiple benefits that pertain to youths’ personal, social and academic life. 

The range of positive outcomes achieved leads to another research question: 
Is it possible to identify the characteristics of ASPs that are associated with 
better outcomes? 



C. What Types of Programs Are More Effeetive? 

Our tliird major finding is that programs that used evidence-based s kill train- 
mg procednres were the only types of programs that were associated with 
positive ontconies. Thirty-nine programs met the criteria for using evidence- 
based procedures in both the content and process of their skill training compo- 
nents; 27 programs did not meet these criteria. Table 4 compares the outcomes 



Table 4: Outcomes for Programs That Did or Did Not Meet Criteria 
Regarding the Use of Evidence-Based Training Approaches 

Outcome Met Criteria Did Not meet Criteria 





ES 


N 


95% Cl 


ES 


N 


95% Cl 


Feelings and Attitudes 


Child self-perceptions 


0.35* 


20 


0.24-0.46 


0.14 


2 


-0.26-0.54 


School bonding 


0.26* 


13 


0.12-0.47 


0.03 


15 


-0.11-0.17 


Indicators of Behavioral Adjustment 
Positive social behaviors 0.30* 


18 


0.19-0.41 


0.06 


17 


-0.07-0.18 


Problem behaviors 


0.26* 


21 


0.16-0.37 


0.07 


21 


-0.04-0.18 


Drug use 


0.22* 


11 


0.07-0.36 


0.03 


16 


-0.09-0.15 


School Performance 


Achievement tests 


0.31 


10 


0.16-0.46 


0.03 


10 


-0.11-0.17 


School grades 


0.24* 


9 


0.07-0.42 


0.05 


16 


-0.07-0.17 


School attendance 


0.15 


9 


-0.01-0.31 


0.07 


12 


-0.05-0.18 



* Denotes mean effect is significantly different from zero at the .05 level 
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social and academic 
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Our major 
hypothesis regarding 
whieh types of 
programs would be 
most sueeessful was 
supported. 




for these two groups of programs. Programs using evidenee-based proeedures 
yielded statistieally signifieant effeets for seven of the eight outeome eategories 
(for all but sehool attendanee). The signifieant mean effeets ranged from 0.22 
for drug use to 0.35 for ehild self-pereeptions. In eontrast, none of the mean 
effeets in any of the eight outeome eategories were signifieant for programs 
that did not use evidenee-based training proeedures. Programs using evidenee- 
based training proeedures produeed mean effeet sizes that were from 2.5 times 
(for ehild self-pereeptions) to 10 times higher in magnitude (for aeademie 
aehievement) than programs not using these proeedures. 

In sum, our major hypothesis regarding whieh types of programs would be 
most sueeessful was supported. Before returning to these data, we sought to 
rule out other possible explanations for these findings. 

D. Ruling out Sotne Possible Rival Explanations for the Findings 
When signifieant findings emerge in a meta-analysis, even if they are based on 
a priori hypotheses, it is important to assess whether other variables eould 
serve as an alternative explanation for the results. 

First, we eompared the effeets in eaeh outeome eategory for studies grouped 
aeeording to eaeh of the following variables: randomization, problems with attri- 
tion, reliability of the outeome measure, presenee of an aeademie eomponent in 
the ASP, aetive parent involvement and, finally, the edueational level of the par- 
tieipants. These proeedures resulted in 48 analyses (six variables erossed with 
eight outeome eategories). Signifieant effeets emerged in only two eases, whieh 
are explained below. These findings suggest that training proeedures were a more 
likely explanation for the positive findings than these other variables. 

Nevertheless, we eontinued with a eonservative analytie strategy. We eon- 
dueted a series of regression analyses and assessed the influenee of other 
potentially important variables by entering these variables ahead of the entry 
of training proeedures. We eondueted hierarehieal weighted least square regres- 
sions using different variables as potential predietors of the effeet sizes for the 
seven outeome eategories for whieh statistieally signifieant effeets were found 
(i.e., for all but sehool attendanee). We eondueted these separate regressions 
beeause the mean effeets did vary aeross the eategories, and a variable eould 
be a signifieant predietor for one type of outeome but not another. 

Our general strategy was first to enter three methodologieal variables into 
eaeh regression in the following order: randomization (yes/no), problems with 
attrition (yes/no) and whether or not the outeome measure was reliable (yes or 
no). Beeause of their potential importanee, we then entered the presenee of an 
aeademie eomponent and aetive parent involvement in that order. Lastly, we 
entered the use of evidenee-based training proeedures. Moreover, we retained 
all the above variables in the regression even if they did not initially aeeount 
for any signifieant varianee in effeets.^ 

Table 5 summarizes the results of these regressions by indieating whieh vari- 

Essentially, this conservative approach attributes any variance shared by training proce- 
dures and the previously entered variables to the latter, effectively making it more difficult for 
training to account for unique variajice. 
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ables emerged as signifieant 
predietors of effeet size for 
eaeh outeome eategory and 
the amount of varianee 
aeeounted for by eaeh sig- 
nifieant predietor. The 
results of these regressions 
provided strong support for 
our hypothesis regarding 
the value of evidenee-based 
proeedures. 

For three outeome eate- 
gories (externalizing behav- 
iors, drug use and sehool bonding), the use of evidenee-based training proeedures 
was the only signifieant predietor, aeeounting for 15% (for externalizing behav- 
ior), 19% (for drug use) or 42% (for sehool bonding) of the varianee in effeets. 
Training proeedures were also signifieant predietors for two other outeome eate- 
gories along with other variables. In these latter eases, randomization predieted 
10% of the varianee in positive soeial behavior and training proeedures predieted 
an additional 9%. Aeademie eomponents predieted 34% of the varianee in aea- 
demie aehievement and training proeedures predieted an additional 15%. No sig- 
nifieant predietors emerged for ehild self-pereeptions or for grades. 



Table 5: Summary of Significant Predictors of Effect 
Size for Different Outcomes 

Amt. of Variance 

Outcome Accounted 



Category 


Predictor(s) 


For 


Feelings and Attitudes 


Child self-perceptions 


None 




School bonding 


Evidence-based training 


42% 


Indicators of Behavioral Adjustment 




Positive social behaviors 


Randomization 


10% 




Evidence-based training 


9% 


Problem behaviors 


Evidence-based training 


15% 


Drug use 


Evidence-based training 


19% 


School Performance 


Achievement tests 


Academic component 


34% 




Evidence-based training 


15% 


School attendance 


None 





E. Other Variables 

We wished to evaluate the influenee of some other variables, but it was impossible 
to do so beeause of missing data. For example, there were too few data on partiei- 
pants’ ethnieity and soeioeeonomie status to examine these variables adequately. 
Setting was strongly assoeiated with the presenee of an aeademie eomponent (i.e., 
sehool-based programs were more likely to offer some form of aeademie assis- 
tanee), so we only entered the latter variable in the regression analysis. 



F. Results at Follow-Up 

Table 6 presents the mean effeets for the 14 reports eontaining follow-up data 
for the different outeome eategories. Unfortunately, there is too little informa- 
tion at follow-up to offer 
eonelusions about the dura- 
bility of ehanges produeed 
by ASPs for the different 
outeomes. For example, 
although the follow-up 
effeet for ehild self-pereep- 
tions is statistieally signifi- 
eant (mean effeet=0.19) it 
is based on only nine stud- 
ies, and this is the highest 
n for any of the eategories. 



Table 6: Mean Effects at Follow- 


Up 




FOR Different Outcomes 








Mean 




95% Confidence 


Outcomes Effect Size 


N 


Interval 


Feelings and Attitudes 








Child self-perceptions 


0.19* 


9 


0.01-0.36 


School bonding 


-0.06 


4 


-0.29-0.16 


Indicators of Behavioral Adjustment 






Positive social behaviors 


0.16 


2 


-0.17-0.49 


Problem behaviors 


0.17 


8 


0.00-0.35 


Drug use 


0.13 


2 


-0.20-0.45 


School Performance 








Achievement tests 


-0.23 


2 


-0.59-0.13 


School grades 


0 


1 


-0.39-0.39 


School attendance 




0 




* Denotes mean effect is significantly different from zero at the .05 level 
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Discussion 



Previous reviews 
have tended tofoeus 
on the aeademie 
outeomes of ASPs 
and have largely 
overlooked other 
possible program 
benefits. 




T his review is the first to evaluate the outeomes aehieved by ASPs that 
seek to promote youths’ personal and soeial skills. Previous reviews 
have tended to foeus on the aeademie outeomes of ASPs and have 
largely overlooked other possible program benefits. This review evaluates a 
large number of ASPs (n=73), and represents the first time many of these 
reports have been serutinized. Sixty pereent of the evaluated reports appeared 
after 2000 and 15% appeared in 2004 or 2005. As a result, this review presents 
an up-to-date perspeetive on a rapidly growing researeh literature. 

We took several steps to inerease the eredibility of the findings. 

1. We searehed earefully and systematieally for relevant reports to obtain a rep- 
resentative sample of evaluations. Although no review is eompletely eompre- 
hensive, we are eonfident that our sample of studies eonstitutes an unbiased 
representation of ASP evaluations that appeared by the end of 2005 and met 
our inelusion eriteria . Signifieantly, we ineluded a large number of unpub- 
lished reports (67% of the total). Ineluding so many unpublished reports pro- 
teets against findings influeneed by publieation bias; that is, published 
reports often produee better results than unpublished reports. 

2. Eaeh ASP was evaluated against its own eontrol group. No single-program 
pre-post evaluations were ineluded beeause, while they often ean produee 
very positive results, they eontain several threats to the internal validity of 
the findings. Twenty-six pereent of the reviewed ASPs used randomized 
experimental designs and the remainder used quasi-experimental designs. 

3. We eonsidered the possible influenee of several methodologieal features of 
the reports, in partieular, the use of randomized designs, problems with attri- 
tion and the reliability of measures used to assess outeomes. With one 
exeeption, these method variables were not signifieant predietors of out- 
eomes. While evaluations of ASPs ean eertainly be improved, eurrent find- 
ings indieate that these methodologieal features did not play a major role in 
the outeome findings. 

While there are several ways that future researeh on ASPs eould be 
improved (and we will diseuss them later in this report), eurrent data support 
the following major eonelusions: 

ASPs that seek to promote personal and soeial skills have an overall positive 
and statistieally signifieant impaet on partieipating youth. Desirable ehanges 
oeeur in three areas: feelings and attitudes, indieators of behavioral adjust- 
ment, and sehool performanee. More speeifieally, there are signifieant inereases 
in youths’ self-pereeptions (i.e., their self-eonfidenee, self-esteem and sense of 
self-effieaey), their bonding to sehool, their positive soeial behaviors, and in 
their sehool grades and performanee on aehievement tests. Furthermore, sig- 
nifieant reduetions oeeur for problem behaviors and drug use. The finding that 
improvements oeeur in multiple domains of young people’s lives offers strong 
support for the value of ASPs. 

An important qualifieation to the above findings is that not all ASPs were 
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effective. As hypothesized, 

ASPs that used evidence- 
based training approaches 
in the skill development 
components of their pro- 
gramming were more effec- 
tive than those that did not. 

In fact, the former pro- 
grams produced significant 
changes on each of the out- 
come areas, whereas the 
latter programs did not lead 
to significant change in any 
of the outcome categories. 

As we discuss below, this 
has important implications 
for future programming. 

It is useful to place the current findings in the context of previous research 
on programs for children and adolescents in general and on ASPs specifically. To 
do so, we use the data on the 39 effective ASPs that used evidence-based train- 
ing procedures. The mean effects achieved in different outcome categories by 
this subset of programs compare very favorably with those obtained by other 
types of youth interventions that assess similar outcomes. Table 7 contrasts the 
results of the current review with those reported in other meta-analyses. 

For these comparisons, we used the findings from other meta-analyses 
regarding universal interventions wherever possible, because the vast majority 
of effective ASPs in our review did not intervene with youth who were already 
experiencing problems. 

Although the number of comparisons varies depending on the outcome, data 
in Table 7 are revealing. In general, effective ASPs produce results equal to or 
better than those produced by other types of interventions for youth (DuBois, 
Holloway, Valentine & Cooper, 2002; Durlak & Wells, 1997; Haney & Durlak, 
1998; Losel & Beelman, 2003; Tobler et al., 2000; Wilson, Gottfredson & 
Najaka, 2001; Wilson, Lipsey & Derzon, 2003). For example, current results for 
externalizing behaviors (0.26) are comparable to or higher than those obtained 
in preventive interventions conducted in schools and other settings (Durlak & 
Wells, 1997; Losel & Beelman, 2003; Wilson et al., 2001; Wilson et al., 2003), 
and those achieved by mentoring programs (DuBois et al., 2002). The mean 
effect for drug use (0.22) is higher than the effect for the most effective school- 
based drug prevention programs (0.15, Tobler et al., 2000) and for other inter- 
ventions assessing this type of outcome (DuBois et al., 2002; Wilson et al., 
2001). Similarly, the results for self-perceptions (0.35) and positive social 
behaviors (0.30) are also higher than or similar to those from other reviews 
(self-perceptions, 0.09, Haney & Durlak, 1998; positive social behaviors, 0.39, 
Losel & Beelman, 2003; and 0.15, DuBois et al., 2002). Finally, the mean effect 
size obtained for effective ASPs on measures of academic achievement (0.31) is 



Table 7: Comparing the Mean Effects from Effective 
After-School Programs to the Results of Other 
Universal Interventions for Children and Adolescents 

Mean Effects 



Outcomes 


Current 

Review 


Other 

Reviews 


Feelings and Attitudes 


Child self-perceptions 


0.35 


0.19' 


School bonding 


0.26 


... 


Indicators of Behavioral Adjustment 
Positive social behaviors 0.30 


0.15‘ 0.39' 


Problem behaviors 


0.26 


0.21" 0.27' 0.09- 


Drug use 


0.22 


0.17' 0.30' 
0.11" 0.05' 0.15' 


School Performance 


Achievement tests 


0.31 


o.ir 0 . 30 ^ 


Grades 


0.24 


— 


School attendance 


0.15 


--- 



Note: Results from other meta-analyses are from outcome categories 
comparable to those in the current review. 

a = Haney & Durlak, 1988; b = DuBois et al., 2002; c = Losel & 
Beelman, 2003; d= Wilson et at, 2003; e = Wilson et at, 2001; 
f = Durlak & Wells, 1997; g = Tobler et at, 2000 
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True no-treatment 
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the highest mean effeet ever obtained in reviews of ASPs (ef. Lauver et al., 
2006), and is similar in magnitude to sueeessful primary prevention programs 
for ehildren and adoleseents (0.30, Durlak & Wells, 1997). 

A mean effeet of 0.31 is a meaningful improvement in aeademie aehieve- 
ment. Although there are variations based on students’ grade levels, an entire 
year of sehooling tends to produee aeademie gains of approximately 0.25 stan- 
dard deviations."* An effeet of 0.31 translates into a pereentile gain of 12% in 
aehievement, whieh is a notieeable gain in test seores. 

Moreover, the eurrent review under-estimates the true impaet of ASPs for at 
least two reasons. One has to do with the nature of the eontrol groups used in 
eurrent evaluations; the seeond has to do with the dosage of the intervention 
reeeived by many program youth. 

Control Groups 

Elsewhere, one of us (Durlak, 1995) has indieated that true no-treatment eon- 
trol groups are a fantasy for sehool-based promotion and prevention programs. 
Most sehools that are in eontrol eonditions nonetheless offer some alternative 
programming. The same situation oeeurs when trying to evaluate ASPs. The 
intent of this review was to eompare outeomes for youth attending a partieular 
ASP to those not attending the program, and we searehed for reports eontain- 
ing the latter type of eomparison group. However, in at least four reports, true 
no-treatment eontrol groups were elearly not obtained (Brooks et al., 1995 ; 
Philliber et al., 2001; Rusehe et al., 1999; Weisman et al., 2003). An appreeia- 
ble number of the “eontrol” youths in these reports were partieipating in alter- 
native ASPs or other types of potentially benefieial out-of-sehool time aetivities. 
For example, Philliber et al. (2001) noted that up to 41% of their eontrols had 
partieipated in other after-sehool program aetivities, and Weisman et al. (2003) 
reported that half of their eontrol group had partieipated in other programs or 
organized groups after sehool. When examining the impaet of promotion or 
prevention programs, it has been reeommended that evaluators monitor the 
types of alternative serviees that are reeeived by eomparison groups, so a truer 
estimate of the impaet of intervention ean be made (Durlak & Dupre, 2007). 

The reeeipt of alternative serviees ean have a substantial influenee on the 
magnitude of effeet sizes. For example, in ehild treatment meta-analyses, when 
analyses are eondueted to determine the differenees between groups reeeiving 
an intervention and eomparison groups reeeiving some alternative serviees, the 
resultant mean effeet sizes are only half as large as those obtained in analyses 
involving intervention groups versus true no-intervention groups (Kazdin & 
Bass, 1989). While limited opportunities exist in some rural areas and inner- 
eity neighborhoods, in most eommunities, youth ean and do partieipate in sev- 
eral different types of aetivities after sehool (e.g., Vandell et al., 2005; Weiss, 
Little & Bouffard, 2005). Therefore, it is likely that eontrol youths’ partieipa- 
tion in alternative out-of-sehool-time aetivities redueed the magnitude of effeet 
in many of the eurrent program evaluations. 



'* The mean effect of 0.31 from the current review is expressed in standard deviation units. 
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Program Dosage 

It is axiomatic that recipients must receive a sufficient dosage of an interven- 
tion for that intervention to have an effect. However, it appears that this did 
not happen in several of the reviewed programs, which may explain the poor 
results obtained in some cases. The duration of each ASP, which is listed in 
Appendix B, does not reflect youths’ attendance patterns. In other words, an 
ASP can last for one full school year (36 weeks), but that does not mean that 
most youth attend regularly. Unfortunately, attendance information was pre- 
sented in different ways across reports, and some reports contained no infor- 
mation on program attendance. When data were available, it was apparent that 
attendance was a problem for several programs. For example, youths’ atten- 
dance ranged from 15% to 26% in 11 programs (Baker & Witt, 1996; Dynarski 
et al., 2004; James-Burdumy et al., 2005; LaFrance et al., 2001; Lauver, 2002; 
Maxfield et al., 2003, both samples; Philliber et al., 2002; Prenovost, 2001, sam- 
ples A, B and G), and between 26 and 50% in three additional situations 
(Chase, 2002, sample A; Prenovost, 2001, sample D; Zief, 2005). 

Moreover, the analyses in some reports indicated that attendance was posi- 
tively related to youth outcomes. This occurred in six of the seven studies that 
examined this issue, although significant differences did not always emerge on 
every outcome measure (Baker & Witt, 1996; Fabiano et al., 2004; Lauver, 
2002; Morrison et al., 2000; Prenevost, 2001; Vandell et al., 2005; Zief, 2005). 
Reviews of other ASPs have also reported a significant positive relationship 
between attendance and positive outcomes (Simkins, Little & Weiss, 2004). 

In sum, the receipt of alternative after-school activities by control groups 
and the low attendance in some programs worked against finding positive out- 
comes. Nevertheless, results of the current review offer clear empirical support 
for the conclusion that well-run ASPs can produce a variety of positive benefits 
that equal or exceed the effects achieved by a variety of promotion and preven- 
tion programs for children and adolescents. As a result, future research and 
policy should support ASPs as an important vehicle for promoting youth devel- 
opment. The next sections discuss several other issues suggested by the cur- 
rent findings. 

Elements of Effeetive ASPs 

Although all the ASPs in our review included as one of their goals the develop- 
ment of youths’ personal and social skills, analyses indicated that only those 
programs that followed four evidence-based training approaches in their pro- 
gram components devoted to skill development produced significant changes 
on any outcomes. Specifically, effective programs had skill-development activi- 
ties that were sequential, active, focused, and explicit. These four features have 
been important in a variety of other skill-oriented interventions for children 
and adolescents, and their value was confirmed again in our analyses with 
respect to ASPs. 

On the basis of these results, we strongly recommend that all ASPs should 
use the evidence-based approaches described in this report. (Other authors 
have mentioned the importance of one or more of these features in ASP pro- 
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grams as well: see Larson & Verma, 1999; Miller, 2003, National Researeh 
Gouneil and Institute of Medieine, 2002.) Not only ean partieipants benefit in 
multiple ways if these eomponents are offered, but sueeess is unlikely if they 
are missing. To improve youths’ personal and soeial skills, programs must 
devote suffieient time to skill enhaneement, be explieit about what they wish 
to aehieve, use aetivities that are eoordinated and sequeneed to aehieve their 
purpose, and require aetive involvement on the part of partieipants. 

Gains in Academic Achievement 

Effective ASPs produced significant improvement in participants’ performance 
on measures of academic achievement (mean effect of 0.31). The single 
strongest predictor of this effect, accounting for 34% of the variance in test 
scores, was the presence of an academic component in the program (either 
tutoring or some form of homework assistance). However, evidence-based 
training procedures emerged as the second-best predictor and accounted for an 
additional 15% of the variance in test scores. There were no other significant 
predictors for this outcome. These particular results merit comment for two 
reasons: (1) the obtained mean effect of 0.31 is nearly twice as large as the 
effects found in the only other meta-analysis of ASPs that has reported signifi- 
cant changes on test scores (Lauer et al., 2006), and (2) the association 
between academic performance and personal and social development is of 
great interest to educators, researchers and policy-makers. Why were the pro- 
grams considered in our review so effective in the academic realm? 

There are several possible explanations for these findings. First, it should be 
noted that only 20 programs collected data on levels of academic achievement, 
so more data from more programs are needed to confirm their generality. 
Second, our results are based on a new set of recent ASP evaluations, only a 
few of which have ever been reviewed previously. Developers of newer ASPs 
may have learned from past reports and experience how to target and improve 
participants’ academic achievement. We did not code the academic compo- 
nents of ASPs, but it is possible that programs that were well-run in terms of 
youths’ personal and social skill development were also well-run in other 
respects, including their academic components. For example, other authors 
have su^ested that gains in academic achievement are more likely to occur if 
staff are well-trained and supervised, provide support and reinforcement to 
youth during learning activities, conduct pre-assessments to ascertain learners’ 
strengths and academic needs, and coordinate their teaching or tutoring with 
school curricula (e.g., American Youth Policy Forum, 2006; Birmingham, 
Pechman, Russell & Mielke, 2005; Southwest Educational Development 
Laboratory, 2006). It would be instructive to analyze the different academic 
components of future ASPs to determine which features are associated with 
better results. 

Third, it comes as no surprise to us that program features related to skill 
development predicted levels of academic achievement. There is now a growing 
body of research indicating that interventions that promote personal and social 
skills also result in improved academic performance (Goleman, 1995; 
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Collaborative for Academic and Social Learning, 2003; Greenberg et al., 2003; 
Weissberg & Greenberg, 1998; Zins et al., 2004). We have obtained a similar 
finding for school-based interventions with an even higher effect size 
(Weissberg, Durlak, Dymnicki & Taylor, 2006). Findings from our regression 
analysis as noted above clearly suggest a relationship between these two 
domains of young people’s functioning. 

Put another way, well-run academic components in ASPs can improve chil- 
dren’s academic achievement, and when they are coupled with well-run skill 
components, students’ achievement can be enhanced even more.^ 

Interventions that recognize the interdependence between youths’ personal 
and social development and their academic development can be very effective. 

What Should We Hold ASPs Accountable For? 

The short answer is that programs should be accountable for achieving the 
goals they have established for themselves, as long as these goals are (1) in line 
with the needs and values of the local community, (2) matter in young people’s 
lives, (3) are realistic and (4) are specific enough so that progress toward goal 
attainment can be assessed. For example, in some communities, parents’ first 
priority might be for an environment that provides for the personal safety of 
their children. Programs are unlikely to be well-attended or successful if they 
do not respond to local needs and preferences. Matching programming to com- 
munity values is also important. For example, several successful programs have 
offered programs that are culturally tailored to the racial and ethnic composi- 
tion of their target population (see Belgrave et al., 2000; Chase, 2000; Fuentes, 
1983; Vincent & Guinn, 2001). New ASPs should secure input from the local 
community before developing and offering programming. Top-down decision- 
making might not result in the correct type of program for the target popula- 
tion. 

Having said that, we would add that current findings indicate youth can ben- 
efit in multiple ways from well-run programs devoted to skill development, so 
we would encourage more ASPs to establish goals related to skill development. 
With the exception of school attendance, the outcomes listed in Table 3 are all 
within reach for such programs, if they are well-run and carefully evaluated. 
That is, improvements can occur in youths’ feelings and attitudes, indicators of 
behavioral adjustment, and in their school performance. This is not to say that 
improvements must occur in all these areas for a particular program to be suc- 
cessful, but benefits should be forthcoming in at least one of these areas. 

Furthermore, we recommend that programs use a logic model to guide their 
efforts. While there are variations, basically, a logic model stresses the need to 



® The empirical connection beVween academic development and personal and social develop- 
ment suggests four possibilities. First, children improve first in academics and their behavior 
subsequently improves. Second, the reverse may occur: skill development may precede aca- 
demic gains. A third possibility is that improvements in academic performance and skills 
occur simultaneously and these positive changes mutually influence each other. Fourth, both 
academic and social development might be promoted by other factors (e.g., a warm and sup- 
portive environment conducive to all types of personal growth). Unfortunately, these alterna- 
tive explanations could not be tested in the current meta-analysis. 
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develop specific program goals, to use activities that can reasonably be expect- 
ed to achieve these goals, and to decide how best to measure goal attainment. 
Moreover, logic models can be used for continual monitoring and program 
improvement, which is essential as programs advance toward their desired 
goals. In other words, logic models can promote accountability and help practi- 
tioners accomplish what they set out to do (see Little, DuPree & Deich, 2002; 
www.gse.harvard.edu/hfrp/projects/afterschool/resources/learning_logic_model. 
html). Community-based organizations are becoming increasingly familiar with 
the components of logic models. 

Are ASPs a Worthwhile Soeial Investment? 

From several vantage points, the results of the current review suggest that 
ASPs are a worthwhile social investment. We have already indicated that par- 
ticipants changed significantly on several types of outcomes and that program 
outcomes compared favorably to those achieved by other recognized and sup- 
ported interventions for youth. Such findings will convince some about the 
value of ASPs. Here we take the discussion further by discussing ASPs from a 
policy- or decision-making perspective. 

Many communities face the same question: Should we offer ASPs? Many fac- 
tors go into making such a decision, but one that could be extremely useful 
would be the ability to estimate how many youth in the community might be 
better off if they participated in ASPs compared to the status quo.*^ It is possible 
to use our findings to make such an estimate. 

To provide some perspective on this issue. Table 8 translates the significant 
mean effect sizes achieved by the programs that used evidence-based training 
procedures into three types of percentages for each outcome category.’ The 
first two numbers refer respectively to the percentages of ASP participants and 



Table 8: An Illustration of the Value-Added Benefits of Effective After-School Programs 

Percentage of 





Program 


Percentage of 


Value-Added 




Participants 


Controls 


Percentage 


Outcomes 


Improving 


Improving 


Benefit* 


Feelings and Attitudes 


Child self-perceptions 


58.75 


41 


42.6 


School bonding 


56.5 


43.5 


29.8 


Indicators of Behavioral Adjustment 


Positive social behaviors 


57.5 


42.5 


35.2 


Problem behaviors 


56.5 


43.5 


29.8 


Drug use 


55.5 


44.5 


24.7 


School Performance 


Achievement tests 


57.75 


42.25 


36.6 


School grades 


56 


44 


27 



*The value-added benefit refers to how many more youth (i.e., what percent) would profit from participating in an ASP 
compared to what is already happening in their community. This number is calculated by dividing the difference in the 
percentages of program and control youth showing improvement by the control percentage rate (e.g., for test scores, 
57.75 minus 42.25 equals 15.50 divided by 42.5 equals 36.6). 




‘ Refer to our earlier discussion that an unknown number of youth are likely to be participat- 
ing in different types of out-of-school time activities. 

^ This is done by converting the effects using Rosenthal and Rubin’s (1982) Binomial Effect Size 
Display and considering the different success rates for ASP participants and controls. 
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controls who improved in each outcome category. The third percentage, called 
a value-added benefit, is the most important. It reflects how many more youth 
could benefit, based on the differences between the improvement rates for pro- 
gram youth and controls. In other words, it indicates how many more youth 
would benefit from an effective program compared to the current state of 
affairs in each community. This type of information should be valuable to poli- 
cy-makers and funders who must decide if investing in an ASP is worthwhile. 

The value-added benefits in the fourth column of Table 8 are noteworthy. 
They range from 27% for school grades to a high of 37% for academic achieve- 
ment. In other words, compared to doing nothing at all, having an effective 
ASP would result in 27% more youth with better grades and 37% more with 
higher achievement test scores. Data for the other categories are also notewor- 
thy: There could be 35% more youth improving in positive social behaviors, 

30% demonstrating less problem behavior, 25% with less drug use, an additional 
30% who feel more connected and bonded to their school, and 43% who feel 
better about themselves and their abilities (child self-perceptions). 

The caveat to the value-added data in Table 8 is that we cannot tell exactly 
which youth would benefit in which category, how much improvement each 
would show, or in how many different ways each child might change. Some 
youth may change a little on only one outcome while others might make signif- 
icant progress in several areas. Nevertheless, the data from Table 8 suggest that 
it is worthwhile to society to have youth participating in ASPs. It seems reason- 
able to support ASP programming that has the potential to be so helpful to so 
many youth in so many different ways. 

Directions for Future Research 

It is common for a review on a rapidly emerging research area such as ASPs to 
generate a series of questions that merit attention in future work. For example, 
the following questions need answers. 

1. Which participants benefit the most from ASPs, and in what areas? 

2. How can attendance and participation be improved for more youth? 

3. How can we create programs that appeal to and benefit youth based on dif- 
ferences in gender, race/ethnicity, age, income status and academic or 
behavioral problems? 

4. What are the long-term benefits of ASPs? 

5. What aspects of program quality are empirically related to youth outcomes 
and should be emphasized in future programming? 

6. How can we best assess the constructs and variables considered by many in 
the field to be important to the success of ASPs, such as participation, 
engagement, program quality, staff composition and competence, and pro- 
gram implementation? 

7. From an ecological perspective (see Weiss, Little & Bouffard, 2005), how do 
child, family, school and neighborhood characteristics lead to consistent and 
active participation in ASPs, and then interact with various program process- 
es and structures to influence youth outcomes? 
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Limitations and Recommendations 

We acknowledge that there are four important limitations in our review and 

several of these lead to recommendations for future research. 

1. Every meta- analysis is limited by the information available in each report. 
Our ability to code each report for potentially important variables was 
restricted by the care with which authors described the goals and proce- 
dures of each program, and their evaluative strategies and findings. For 
example, some programs may have used the four characteristics of evidence- 
based training, but did not describe their approach in a way that made this 
clear. Many reports lacked data on the racial and ethnic composition or the 
socioeconomic status of participants, so we could not relate outcomes to 
these characteristics. At a minimum, future program evaluations should pro- 
vide complete information on the demographic characteristics of partici- 
pants and, if pertinent, their prior academic achievement and any present- 
ing problems they might have. The goals and procedures relevant to each 
program component should be specified and described, and data on atten- 
dance patterns and levels of participation should be included. Reliable out- 
come measures should be used and, whenever possible, data should be col- 
lected via multiple methodologies (e.g., from school records, questionnaires 
and observations) and from multiple informants (e.g., youth, parents, teach- 
ers and ASP staff). Whenever possible, randomized designs are preferred 
because they strengthen the confidence researchers and policy-makers can 
have in the findings. 

It would also be useful to conduct outcome analyses on subgroups of par- 
ticipants, grouped on the basis of participant and family characteristics, to 
determine how well programs are serving different youth. Ideally, mediation- 
al analyses would be performed to assess the active ingredients of each pro- 
gram. This could help answer questions about what program features lead to 
what initial changes, which are then connected with subsequent changes in 
youth functioning. For example, if youths’ skills improve first, does that lead 
to other positive outcomes (improved academic achievement, fewer behav- 
ioral problems, and so on), or are other factors responsible for the positive 
outcomes? Once again, using a logic model can help evaluators determine 
“what leads to what.” 

2. Our findings on the importance of training approaches are just the beginning 
of the effort to open the “black box” of ASPs and understand the structures 
and processes that constitute an effective program. It is possible that other 
variables play an important role in the effectiveness of ASPs, and they may 
overlap with or supersede the constructs we examined. One feature that 
comes to mind, and has been emphasized elsewhere (Granger & Kane, 2004; 
High/Scope Educational Research Foundation, 2005), is program quality. It 
is hard to deny that quality matters, but it is not clear how quality should be 
defined and assessed. Several approaches to assessing the quality of youth 
programming have now appeared. These assessments examine a range of 
program operations that include structure, staff behavior, youth engagement 
and initiative, and issues related to implementation (see Birmingham et al.. 
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2005; Forum for Youth Investment, 2006; Miller, 2003; Vandell et al., 2004; 
www.youth.highseope.org). Assessing program quality aeross a range of 
dimensions and relating these to outeomes ean provide an empirieal basis 
for understanding the proeesses within ASPs that lead to different results. As 
researeh on this topie aeeumulates, it will be possible to develop a elearer 
understanding of what eonstitutes a high-quality program. 

3. Very few reports have eolleeted any follow-up data, so we eannot offer any 
eonelusions about the long-term effeets of ASPs. Hopefully, future reports 
will eolleet follow-up information to determine the durability of any gains 
derived from program partieipation. 

4. Although the initial database of studies seems suffieiently large (66 studies 
with post-data), dividing studies first aeeording to outeome eategories and 
then aeeording to other potentially important variables redueed the statisti- 
eal power of the analyses. Therefore, the failure to obtain statistieally signifi- 
eant findings for the variables examined here should be viewed eautiously 
(e.g., edueational level, family involvement). As more ASP evaluations 
appear, researehers will have more power to deteet the influenee of poten- 
tially important variables. 

Notwithstanding the above issues, we believe our review offers good empiri- 
eal support for the value of ASPs. We hope it will stimulate more interest in 
investigating and understanding how these programs affeet youth, and what 
ean be done to enhanee their effeetiveness. 



As research on this 
topic accumulates, it 
will be possible to 
develop a clearer 
understanding of 
what constitutes a 
high-quality 
program. 
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Appendix B 



Table B1: Descriptive Information on Reviewed Reports with Effect Sizes at Post, N=66 



Author 


Year 


Academic 

Program Name Component 


Active Parent 
Involvement 


Four S.A.F.E. 
Components 


Study 

Effect Sizes* 


Astroth & Haynes 


2002 


4-H Clubs 


no 


no 


no 


0.25 


Baker et al. 


1995 


South Baltimore Youth Center 


no 


no 


no 


0.74 


Baker & Witt 


1996 




no 


no 


yes 


0.14 


Belgrave et al 


2000 




no 


no 


yes 


0.36 


Bergin et al 


1992 


Hilltop Emergent Literacy Project 


no 


no 


no 


0.57 


Bissell et al. 


2002 


YS-CARE 


no 


no 


yes 


0.09 


Brooks 


1995 


LA’s BEST Final Evaluation Report 


yes 


no 


no 


0.31 


Chase 


2000 


Hmong -American Partnership Program 


yes 


yes 


yes 


0.48 


Chase 


2000 


Hmong -American Partnership Program 


yes 


yes 


yes 


-0.05 


Dynarski et al 


2004 


National Evaluation 
of 21st Century Learning Centers 


yes 


no 


No 


0 


Fabiano et al 


2005 


Citizen Schools 


no 


no 


yes 


0.23 


Foley & Eddins 


2001 


Virtual Y 


no 


no 


no 


0.09 


Fuentes 


1983 


Hispanic After School Program 


no 


no 


yes 


0.17 


Gottfredson et al 


2004 


Maryland After School Programs 


yes 


no 


no 


0.07 


Grenawalt et al 


2005 


4-H Clubs 


no 


no 


yes 


0.53 


Hudley 


1999 


Anger Control Program 
in Boys and Girls Clubs 


no 


no 


yes 


0.39 


James-Burdumy et al 


2005 


National Evaluation 
of 21st Century Learning Centers 


yes 


no 


no 


-0.01 


LaFrance Associates 


2001 


Bayview Safe Haven 


yes 


yes 


yes 


0.59 


Lauver 






no 


no 


no 


0.13 


LoSciuto & Hilbert 


1999 


Woodrock Development Program 


yes 


yes 


yes 


0.16 


Mahoney et al 


2005 




no 


no 


yes 


0.24 


Mason & Chuang 


2001 


Kuumba Kids 


no 


no 


yes 


0.64 


Maxfield et al 


2003 


Quantum Opportunity Demonstration Project: 
Well-implemented site 


yes 


no 


no 


0.01 


Maxfield et al 


2003 


Quantum Opportunity Demonstration Project: 
Sites with implementation 


yes 


no 


no 


0.11 


McClanahan et al 


2004 


Summer Career Exploration Program 


no 


no 


yes 


0.76 


Monsaas 


1994 


Project Emerge: 4th grade cohort 


yes 


no 


yes 


0.85 


Monsaas 


1994 


Project Emerge: 5th and 6th graders 


yes 


no 


yes 


0.66 


Morrison et al. 






yes 


yes 


yes 


0.05 


Neufeld et al 






yes 


no 


yes 


0.54 


Oyserman et al. 


2002 


School-to-Jobs 


no 


no 


yes 


0.31 


Philliber et al. 


2001 


Children’s Aid Society Carerra Program 


yes 


no 


yes 


0.14 


Phillips 






yes 


no 


yes 


0.98 


Pierce & Shields 


1998 


Be A Star Program 


no 


yes 


yes 


0.62 


Prenovost 


2001 




no 


no 


no 


-0.05 


Prenovost 


2001 




no 


no 


no 


0.04 


Prenovost 


2001 




no 


no 


no 


0.06 


Prenovost 


2001 




no 


no 


no 


0 



* Effects are the average of all outcomes at post within each report and those in parentheses are similarly averaged at follow-up. 
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Table B1 (cont.): Descriptive Information on Reviewed Reports with Effect Sizes at Post, N=66 



Author 


Year 


Academic 

Program Name Component 


Active Parent 
Involvement 


Four S.A.F.E. 
Components 


Study 

Effect Sizes* 


Ross 


1992 


adept Drug & Alcohol 
Community Prevention Program 


no 


no 


yes 


0.18 


Rusche et al 


1999 


Club Hero 


yes 


yes 


yes 


-0.03 


Schinke et al. 


1992 


smart Moves Program in Boys and Girls Club 


no 


no 


yes 


0.07 


Schinke et al. 


1989 




no 


no 


yes 


0.29 


Smith et al 


1979 




no 


no 


yes 


0.35 


Smoll et al 


1993 




no 


no 


yes 


0.32 


St. Pierre et al. 


2001 


SMART Moves Program in Boys and Girls Club 


yes 


yes 


yes 


0.39 


St. Pierre et al. 


1992 


Stay SMART 


no 


no 


yes 


0.38 


St. Pierre et al. 


1997 


Family Advocacy Network (FAN) club 


no 


no 


yes 


0.08 


Tucker & Herman 


2002 


Model Program 


yes 


yes 


yes 


0.26 


Vandell et al 


2004 




yes 


no 


yes 


0.27 


Vandell et al 


2005 




yes 


no 


yes 


0.07 


Vincent & Guinn 


2001 


Colonia Program 


no 


no 


yes 


0.67 


Weisman et al 


2003 


Maryland After-School 
Community Grant Program 


yes 


no 


no 


-0.08 


Weisman et al 


2001 Maryland After School Grant Program (MASP): 
2001-2002 school year 


yes 


no 


no 


0.11 


Weisman et al 


2001 


MASP 2001-2002 


yes 


no 


no 


0.19 


Weisman et al 


2001 


MASP 2001-2002 


no 


no 


no 


0.10 


Weisman et al 


2001 


MASP 2001-2002 


yes 


no 


yes 


0.43 


Weisman et al 


2001 


MASP 2001-2002 


yes 


no 


yes 


0.31 


Weisman et al 


2001 


MASP 2001-2002 


yes 


yes 


yes 


0.3 


Weisman et al 


2001 


MASP 2001-2002 


no 


no 


no 


0.16 


Weisman et al 


2001 


MASP 2001-2002 


no 


no 


no 


0.16 


Weisman et al 


2001 


MASP 2001-2002 


no 


no 


No 


0 


Weisman et al 


2001 


MASP 2001-2002 


no 


no 


No 


-0.15 


Weisman et al 


2001 


MASP 2001-2002 


no 


no 


Yes 


0.03 


Weisman et al 


2001 


MASP 2001-2002 


no 


no 


No 


-0.16 


Weisman et al 


2001 


MASP 2001-2002 


yes 


no 


No 


0.13 


Weisman et al 


2001 


MASP 2001-2002 


no 


no 


No 


-0.04 


Zief 


2005 




no 


yes 


no 


-0.01 



* Effects are the average of all outcomes at post within each report and those in parentheses are similarly averaged at follow-up. 
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Table B2: Descriptive Information on Reviewed Reports with Follow-Up Effect Sizes, N=14 

Academic Active Parent Evidence-Based Study 



Author 


Year 


Program Name 


Component 


Involvement 


Training 


Effect Sizes* 


Chase 


2000 


Hmong-American 
Partnership Program 


yes 


yes 


yes 


0.08 


Fabiano et al 


2005 


Citizen Schools 


no 


no 


yes 


0.11 


Fuentes 


1983 


Hispanic After School Program 


no 


no 


yes 


-0.27 


Hahn et al 


1994 


Quantum Opportunity 
Demonstration Project 


yes 


no 


no 


0.52 


Hahn et al 


1994 


Quantum Opportunity 
Demonstration Project 


yes 


no 


no 


0.26 


Hahn et al 


1994 


Quantum Opportunity 
Demonstration Project 


yes 


no 


no 


0.38 


Hahn et al 


1994 


Quantum Opportunity 
Demonstration Project 


yes 


no 


no 


0.34 


Huang 


2004 


LA’s BEST (follow-up) 


yes 


no 


no 


-0.13 


Huang 


2005 


LA’s BEST (follow-up) 


yes 


no 


no 


0.10 


LaFrance Associates 


2001 


Bayview Safe Haven 


yes 


yes 


yes 


0.61 


McClanahan et al. 


2004 


Summer Career 
Exploration Program 


no 


no 


yes 


0.01 


Rusche et al 


1999 


Club Hero 


yes 


yes 


yes 


0.05 


Schinke et al 


1989 




no 


no 


yes 


0.29 


Smoll et al 


1993 




no 


no 


yes 


0.62 



* Effects are the average of all outcomes within the report at follow-up. 



Program Notes 



A. General Approaeh in Evalnating Programs 
in Eaeh Report 

This review evaluates findings for a total of 73 after- 
school programs, described in 55 sources. What 
accounts for the higher number of evaluated pro- 
grams compared to the smaller number of sources? 

In several cases, a report contained evaluations of 
more than one after-school program. Whenever a 
source evaluated multiple after-school programs that 
differed in their operational features, and as long as 
each program had its own control group, results were 
evaluated separately for each program. Keeping these 
controlled programs separate is preferable to collaps- 
ing different types of programs and participants 
together. 

More than one after-school program is evaluated in 
the following reports. Four different programs were 
evaluated in two sources (Hahn, Leavitte & Aaron, 
1994; Prenovost, 2001); Weisman, Soule & Womer 
(2001) yielded data on 14 different programs; 
Maxfield, Schirm & Rodriguez-Planas (2003), 



Monsaas (1994), and Vandell et al. (2005) each pro- 
duced data on two programs; and, finally, Chase 
(2000) contained information on three programs (two 
at post, and one only at follow-up). 

Of the 73 programs, there are data at post for 66 
programs. Seven reports contain data at follow-up 
only, while seven programs offer data at both post 
and follow-up. Because of the small number of stud- 
ies, the follow-up data are not analyzed statistically, 
but are presented in Table 6 for inclusiveness. One 
immediate implication is that more information is 
needed on the durability of the impact from after- 
school programs. 



B. Notes on Individual Reports 

Most reports contained information on a single inter- 
vention and control group. Those reports in which 
decisions had to be made regarding which samples or 
intervention conditions to use are discussed in alpha- 
betical order according to author. 

For LA’s BEST Program (Brooks et al., 1995), data 
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were evaluated for students with “at least two years of 
program partieipation” beeause that was the eriterion 
used in this report for partieipating youth. 

Follow-up effeets for different eohorts of youth in 
LA’s BEST program are eontained in Huang et al. 
(2004) and Huang et al. (2005). The former report is 
a seven-year follow-up of the eohort of third- to fifth- 
graders who had partieipated from 1994 to 1995. In 
the latter ease, data from students with three years of 
partieipation were evaluated in a four-year follow-up 
of sixth- to ninth-graders who had originally partiei- 
pated from 1998 to 1999. 

The national evaluation of 21st Century Learning 
Centers has been deseribed in several reports. 
Evaluations have been done for the randomized trial 
of programs for elementary students and the quasi- 
experimental trial of programs for middle sehool stu- 
dents, and at different time points. 

We used the data from Dynarski et al. (2004) to 
assess the benefits of two years of partieipation in the 
21st Century Learning Center Programs for middle 
sehool students. 

We used the data from James-Burdumy et al. 

(2005) to assess the benefits of two years of partieipa- 
tion in the 21st Century Learning Center Programs 
for elementary students. 

The evaluation of the Quantum Opportunity Pilot 
Program (Hahn et al., 1994) provides data at follow- 
up only for three separate eohorts. This is another 
nationally reeognized program. 

The Woodroek Development Program (LoSeiuto et 
al., 1999) does eontain some aetivities that took plaee 
during sehool hours, but this is a multifaeeted pro- 
gram with several after-sehool eomponents that we 
felt merited inelusion. Moreover, this program has 
reeeived national reeognition as a youth development 
program. 

Maxfield et al. (2003) evaluated a large-seale 
demonstration projeet of the Quantum Opportunities 
Program, following the sueeess of the pilot Quantum 
Program reported in Hahn et al. (1994). For the 
Maxfield et al. report, we ealeulated separate effeets 
for the independent sites that were implemented well 
(namely, Philadelphia and Yakima) and those with 
implementation problems (Forth Worth, Cleveland, 
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Washington, D.C., Houston and Memphis). The 
results did differ between these two sets of sites. 

Projeet Emerge deseribed by Monsaas (1994) is the 
only program in our review that operated before 
sehool started in the morning. We ealeulated separate 
effeets for the fourth-grade eohort and for the fifth- 
and sixth-grade eohorts beeause they differed in 
terms of program duration and aetivities. Eaeh has its 
own eontrol group. 

For the Carerra Program, we ealeulated effeets 
based on the data for the full study sample that were 
available in the unpublished report (Philliber, Kaye & 
Herlin, 2001) instead of the information on the small- 
er sample that was reported in the publieation by 
Philliber, Kaye, Herring & West(2002). 

For Ross et al. (1992), only data from the more 
eomplete intervention that ineorporated self-esteem 
enhaneement aetivities were evaluated. 

Smith, R., Smoll & Curtis (1979) deseribed a reere- 
ational sports program for boys that trained little 
league eoaehes on strategies to promote self-eonfi- 
denee and better peer relationships in youth. We 
ineluded it along with another investigation by the 
same researeh group (Smoll, Smith, Barnett & 
Everett, 1993) beeause these reports were the only 
eontrolled after-sehool outeome studies we eould find 
that foeused speeifieally on reereational aetivities. 
Many after-sehool programs offer reereational aetivi- 
ties, so these two studies provide important data on 
how these aetivities ean be used to promote youths’ 
soeial development. 

In the St. Pierre et al. (1997) report, we only eoded 
data for the most eomprehensive eondition, that is, 
the SMART MOVES plus FAN Club eondition that 
involved parents. 

The report by Vandell et al. (2005) presents an 
innovative way to assess the effeets of after-sehool 
aetivities. The authors examined the extent to whieh 
youth partieipated in formal after-sehool programs 
and how they spent their after-sehool time in other 
ways, sueh as sehool-based extraeurrieular aetivities, 
eoaehed sports, various lessons, being at home alone 
or with siblings with no adults present, and hanging 
out with peers. They then used eluster analysis to 
identify four groups of youth: 
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1. High program/high activity group — i.e., high in for- 
mal after-school program participation and high 
involvement in other struetured after-sehool aetivi- 
ties 

2. High program/low aetivity group — i.e., high in after- 
sehool program partieipation but low in other 
struetured after-sehool aetivities 

3. Low supervision group — low partieipation in formal 
after-sehool programs and high amounts of time in 
unsupervised settings (espeeially hanging out with 
friends) 

4. Supervised at home group — low on all, suggesting 
they are at home and under the supervision of 
adults 

We used the low supervision group as a eontrol 
eondition and eombined the two high program aetivi- 
ty groups to evaluate the elementary and middle 
sehools students as separate intervention groups. 

Eaeh had a respeetive eontrol group. 

Admittedly, the intervention samples from this 
report evaluate the effeets of youth partieipation in 



more than just formal after-sehool programs, but the 
eomparison to the low supervision youth is informa- 
tive. It is partieularly useful to know if youth who 
partieipated under the supervision of adults in after- 
sehool aetivities fare better than those who are basi- 
eally “on their own” when sehool lets out. 

Weisman, Soule & Womer (2001) generated find- 
ings for 14 different after-sehool programs. The 
results from this study have been published, but in a 
very different format (Gottfredson, Gerstenblith, 
Soule, Womer & Lu, 2004). In the published study 
the authors eombined data for 14 programs serving 
either younger (grades 4-5; n=6 programs) or older 
youth (grades 6-8; n=8 programs). However, beeause 
some of the 14 programs used randomized designs 
and varied in their programming, and eaeh had its 
own eontrol group, we used data from the unpub- 
lished report for our analyses. As a result, we evalu- 
ated 14 separate after-sehool programs, whieh varied 
in experimental design, program approaeh and out- 
eomes. 
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Appendix C; Tables Showing Effeet Size by Outeome Category 



Table Cl: Effect Size by Outcome Category: Child Self-Perceptions 






Author 


Program Name 


Outcome Name 


Effect Size* 


Baker & Witt 




self-esteem 


0.38 


Belgrave et al. 




self-esteem, racial identity 


0.38 


Brooks 


LA’s BEST Final Evaluation Report 


self-concept 


0.29 


Chase (A) 




cultural pride 


(1.18) 


Chase (B) 


Hmong-American Partnership Program 


cultural pride 


0.69 


Chase (C) 


Hmong-American Partnership Program 


cultural pride 


0.01 


Fuentes 


Hispanic After School Program 


self concept scale 


0.56 (-0.27) 


Hahn et al. (A) 


Quantum Opportunity Demonstration Project 


positive feelings 


(0.65) 


Hahn et al. (B) 


Quantum Opportunity Demonstration Project 


positive feelings 


(0.41) 


Hahn et al. (C) 


Quantum Opportunity Demonstration Project 


positive feelings 


(0.33) 


Hahn et al. (D) 


Quantum Opportunity Demonstration Project 


positive feelings 


(0.44) 


Huang et al. (2004) 


LA’s BEST 


academic confidence 


(-0.14) 


LoSciuto & Hilbert 


Woodrock Development Program 


Harter self-esteem 


0.13 


Mason & Chuang 


Kuumba Kids 


self-esteem, self-reliance. 


0.37 


McClanahan 


Summer Career Exploration Program 


self-esteem 


(0) 


Monsaas 


Project Emerge: 4th grade cohort 


self-esteem 


0.48 


Monsaas 


Project Emerge: 5th and 6th grade cohort 


self-esteem 


0.17 


Morrison et al. 




academic self-concept 


0.14 


Table Cl (continued) 
Neufeld et al. 




locus of control; feeling in control 


1.08 


Oyserman et al. 


School-to-Jobs 


balanced selves, self-concept 


0.28 


Phillips 




self-esteem 


1.21 


Pierce St Shields 


Be A Star Program 


self-esteem, emotional awareness/self-control 


0.51 


Ross 


ADEPT Drug Si Alcohol 
Community Prevention Program 


F- egotism (T), A- self-esteem (T)(P) 


0.24 


Rusche et al. 


Club Hero 


self-esteem 


-0.05 (-0.05) 


Smith et al. 




self-esteem 


0.56 


Smoll et al. 




self-esteem 


0.20 


Vandell et al. 




self-efficacy 


0 


Vincent St Guinn 


Colonia Program 


self-esteem, perception of health locus of control 0.67 


Zief 




self-esteem 


-0.11 


* Effects contained in parentheses are at follow-up. 







CASEL 



40 



Collaborative for Academic, Social, and Emotional Learning 



2007 




Table C2: Effect Size by Outcome Category: School Bonding 






Author 


Program Name 


Outcome Name 


Effect Size* 


Bissell et al. 


YS-CARE 


school bonding 


0.04 


Brooks 


LA’s BEST Final Evaluation Report 


attitudes toward school 


0.26 


Chase (A) 




school bonding 


(-0.42) 


Chase (B) 


Hmong-American Partnership Program 


school bonding 


0.02 


Chase (C) 


Hmong-American Partnership Program 


school bonding 


0.12 


Dynarski et al. 


National Evaluation of 21st Century Learning Centers 


educational aspirations 


0.03 


Huang et al. (2004) 


LA’s BEST 


school bonding 


(-0.18) 


Lauver 


college aspirations; commitment to education 


0.35 


Mason & Chuang 


Kuumba Kids 


attitude toward school 


-0.14 


McClanahan 


Summer Career Exploration Program 


educational aspirations 


(-0.08) 


Morrison et al. 




bonding to school 


0.29 


Oyserman et al. 


School-to-Jobs 


concern about school, bonding 


0.31 


Pierce St Shields 


Be A Star Program 


bonding to school 


1 


Rusche et al. 


Club Hero 


school bonding 


-0.04 (0.22) 


St. Pierre et al. (1992) 


SMART Moves Program in Boys and Girls Club 


child survey school bond 


0.51 


Weisman et al. (A) (2001) 


Maryland After School Grant Program (MASP): 
2001-2002 school year 


bonding to school 


0.02 


Weisman et al. (B) (2001) 


MASP 2001-2002 


bonding to school 


0.03 


Weisman et al. (C) (2001) 


MASP 2001-2002 


bonding to school 


0.4 


Weisman et al. (D) (2001) 


MASP 2001-2002 


bonding to school 


0.04 


Weisman et al. (E) (2001) 


MASP 2001-2002 


bonding to school 


0.23 


Weisman et al. (F) (2001) 


MASP 2001-2002 


bonding to school 


0.18 


Weisman et al. (G) (2001) 


MASP 2001-2002 


bonding to school 


0.07 


Weisman et al. (H) (2001) 


MASP 2001-2002 


bonding to school 


-0.01 


Weisman et al. (1) (2001) 


MASP 2001-2002 


bonding to school 


-0.1 


Weisman et al. (J) (2001) 


MASP 2001-2002 


bonding to school 


-0.17 


Weisman et al. (K) (2001) 


MASP 2001-2002 


bonding to school 


0.03 


Weisman et al. (L) (2001) 


MASP 2001-2002 


bonding to school 


-0.41 


Weisman et al. (M) (2001) 


MASP 2001-2002 


bonding to school 


0.31 


Weisman et al. (N) (2001) 


MASP 2001-2002 


bonding to school 


0 


Weisman et al. (2003) 


MASP 2002-2003 


commitment to education 


-0.21 


Zief 




bonding to school 


-0.12 


* Effects contained in parentheses are at follow-up. 
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Table C3: Effect Size by Outcome Category: Positive Social Behaviors 






Author 


Program Name 


Outcome Name 


Effect Size* 


Astroth & Haynes 


4-H Clubs 


helping others, leadership positions 


0.23 


Baker et al. 


South Baltimore Youth Center 


pro-social behavior 


-0.13 


Bergin 


Hilltop Emergent Literacy Project 


classroom behavior- social skills 


0.67 


Bissell et al. 


YS-CARE 


citizenship (social skills) 


0.15 


Dynarski et al. 


National Evaluation of 21st Century Learning Centers 


peer interactions, social skills 


0 


Gotfredson, Soule SiCross 




social skills 


0.23 


Grenawalt et al. 


4-H Clubs 


getting along with others 


0.45 


Huang et al. (2004) 


LA’s BEST 


social competencies 


(0.02) 


Hudley 


Anger Control Program in Boys and Girls Clubs 


social skills (P), SSRS social skills (T) 


0.41 


James-Burdumy 


National Evaluation of 21st Century Learning Centers 


social skills 


-0.1 


Mahoney 




peer popularity, friendships 


0.24 


Mason & Chuang 


Kuumba Kids 


leadership, social skills, adaptability, 
interpersonal relations 


0.85 


Morrison et al. 




assertiveness, cooperation, social skills 


0.15 


Pierce St Shields 


Be A Star Program 


social skills 


0.44 


Schinke et al. 




refusal skills 


0.34 (0.35) 


Smith et al. 




peer relationships 


0.34 


Smoll et al. 




peer relationships 


0.33 


St Pierre et al. (1997) 


Family Advocacy Network (FAN) club 


refusal and social skills 


0.15 


St. Pierre et al. (1992) 


SMART Moves Program in Boys and Girls Club 


TASS problem solving, refusal skills, 
courteousness, or ethical behavior 


0.35 


Vandell et al. (A) 




social skills (staff) 


0.33 


Vandell et al. (B) 




social skills (staff) 


0 


Weisman et al. (A) (2001) 


Maryland After School Grant Program (MASP): 
2001-2002 school year 


social skills 


0.03 


Weisman et al. (B) (2001) 


MASP 2001-2002 


social skills 


0.47 


Weisman et al. (C) (2001) 


MASP 2001-2002 


social skills 


0.3 


Weisman et al. (D) (2001) 


MASP 2001-2002 


social skills 


0.59 


Weisman et al. (E) (2001) 


MASP 2001-2002 


social skills 


0.12 


Weisman et al. (F) (2001) 


MASP 2001-2002 


social skills 


0.39 


Weisman et al. (G) (2001) 


MASP 2001-2002 


social skills 


0.15 


Weisman et al. (H) (2001) 


MASP 2001-2002 


social skills 


-0.39 


Weisman et al. (1) (2001) 


MASP 2001-2002 


social skills 


-0.16 


Weisman et al. (J) (2001) 


MASP 2001-2002 


social skills 


-0.6 


Weisman et al. (K) (2001) 


MASP 2001-2002 


social skills 


0.32 


Weisman et al. (L) (2001) 


MASP 2001-2002 


social skills 


0.08 


Weisman et al. (M) (2001) 


MASP 2001-2002 


social skills 


0.61 


Weisman et al. (N) (2001) 


MASP 2001-2002 


social skills 


-0.35 


Weisman et al. (2003) 


Maryland After-School Community Grant Program 


social skills 


0.1 


* Effects contained in parentheses are at follow-up. 
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Table C4: Effect Size by Outcome Category: Problem Behaviors 






Author 


Program Name 


Outcome Name 


Effect Size* 


Astroth & Haynes 


4-H Clubs 


criminal activity 


0.24 


Baker & Witt 




conduct (child, teen, parent) 


0 


Baker et al. 


South Baltimore Youth Center 


delinquent behavior 


1.10 


Brooks 


LA’s BEST Final Evaluation Report 


behavior change 


0.35 


Dynarski et al. 


National Evaluation of 21st Century 
Learning Centers 


school suspensions, behavior problems 
at school, negative behavior composite 


-0.03 


Fabiano 


Citizen Schools 


suspensions 


0.03 


Fuentes 


Hispanic After School Program 


mental health referrals 


0.31 


Gotfredson, Soule SiCross 




suspension, expulsion, delinquency 


0.07 


Hahn et al. (A) 


Quantum Opportunity Demonstration Project 


trouble with police 


(0.5) 


Hahn et al. (B) 


Quantum Opportunity Demonstration Project 


trouble with police 


(0.03) 


Hahn et al. (C) 


Quantum Opportunity Demonstration Project 


trouble with police 


(0.53) 


Hahn et al. (D) 


Quantum Opportunity Demonstration Project 


trouble with police 


(0.68) 


Huang et al. (2004) 


LA’s BEST 


trouble at school, fighting 


(-0.06) 


Hudley 


Anger Control Program in Boys and Girls Clubs 


problem behaviors (T&P), SSRS aggression 


0.37 


James-Burdumy 


National Evaluation of 21st Century 
Learning Centers 


behavior problems at school, 
suspensions, negative behavior composite 


-0.04 


LaFrance Associates 


Bayview Safe Haven 


arrests, delinquent behavior, 
school suspensions 


0.79 (0.60) 


Lauver 




classroom conduct 


0.2 


LoSciuto & Hilbert 


Woodrock Development Program 


aggression 


0.19 


Mason & Chuang 


Kuumba Kids 


attention problems (T) 


0.85 


Maxfield et al. (A) 


Quantum Opportunity Demonstration Project: 
Well implemented sites 


ever arrested 


0 


Maxfield et al. (B) 


Quantum Opportunity Demonstration Project: 
Sites with implementation problems 


ever arrested 


0 


McClanahan 


Summer Career Exploration Program 


criminal records 


(0) 


Monsaas (A) 


Project Emerge: fourth-grade cohort 


discipline referral 


0.97 


Monsaas (B) 


Project Emerge: fifth- and sixth-grade cohort 


discipline referral 


1.35 


Morrison et al. 




acting out 


-0.22 


Oyserman et al. 


School-to-Jobs 


avoiding trouble, school discipline 


0.22 


Philliber et al. 


Children’s Aid Society Carerra Program 


delinquent acts, aggression, & violence 


0.05 


Ross 


ADEPT Drug & Alcohol Community 
Prevention Program 


risky behaviors, impulsivity, acting out 


0.34 


Rusche et al. 


Club Hero 


problem behavior 


-0.02 (-0.2) 


Schinke et al. 


SMART Moves Program in Boys and Girls Club 


juvenile crime activity 


0.07 


Vandell et al. (A) 




misconduct, aggressive 


0.30 


Vandell et al. (B) 




misconduct, aggressive 


0.16 


Weisman et al. (A) (2001) 


Maryland After School Grant Program (MASP): 
2001-2002 school year 


delinquency 


0.29 


Weisman et al. (B) (2001) 


MASP 2001-2002 


delinquency 


0.11 


Weisman et al. (C) (2001) 


MASP 2001-2002 


delinquency 


0.01 


* Effects contained in parentheses are at follow-up. 
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Table C4 (cont.): Effect Size by Outcome Category: Problem Behaviors 






Author 




Program Name 


Outcome Name 


Effect Size* 


Weisman et al. (D) (2001) 




MASP 2001-2002 


delinquency 


0.74 


Weisman et al. (E) (2001) 




MASP 2001-2002 


delinquency 


0.53 


Weisman et al. (F) (2001) 




MASP 2001-2002 


delinquency 


0.53 


Weisman et al. (G) (2001) 




MASP 2001-2002 


delinquency 


0.09 


Weisman et al. (H) (2001) 




MASP 2001-2002 


delinquency 


-0.13 


Weisman et al. (1) (2001) 




MASP 2001-2002 


delinquency 


-0.08 


Weisman et al. (J) (2001) 




MASP 2001-2002 


delinquency 


-0.29 


Weisman et al. (K) (2001) 




MASP 2001-2002 


delinquency 


0.1 


Weisman et al. (L) (2001) 




MASP 2001-2002 


delinquency 


-0.27 


Weisman et al. (M) (2001) 




MASP 2001-2002 


delinquency 


-0.28 


Weisman et al. (N) (2001) 




MASP 2001-2002 


delinquency 


0.46 


Weisman et al. (2003) 


Maryland After-School Community Grant Program 


rebellious behavior, delinquency 


-0.20 


Zief 






discipline at school, behavior problems 


0.05 


* Effects contained in parentheses are at foliow-up. 









Table C5: Effect Size by Outcome Category: Drug Use 






Author 


Program Name 


Outcome Name 


Effect Size* 


Astroth & Haynes 


4-H Clubs 


drug use 


0.24 


Baker et al. 


South Baltimore Youth Center 


drug or alcohol use 


0.82 


Dynarski et al. 


National Evaluation of 21st Century Learning Centers 


drug use 


-0.02 


Gotfredson, Soule & Cross 




last month drug use 


0.05 


LoSciuto & Hilbert 


Woodrock Development Program 


drug use 


0.18 


Maxfield et al. (A) 


Quantum Opportunity Demonstration Project: Well implemented sites 


drug use or abuse 


0 


Maxfield et al. (B) 


Quantum Opportunity Demonstration Project: 
Sites with implementation problems 


drug use or abuse 


0 


Philliber et al. 


Children’s Aid Society Carerra Program 


drug use 


0.02 


Rusche et al 


Club Hero 


drug use 


0 (0.05) 


Schinke et al. 




drug use 


0.23 (0.23) 


St Pierre et al. (1997) 


Family Advocacy Network (FAN) club 


drug use 


0 


St Pierre et al. (1992) 


Stay SMART 


drug use 


0.38 


Vandell et al. (B) 




drug use 


0.34 


Weisman et al. (A) (2001) 


Maryland After School Grant Program (MASP): 
2001-2002 school year 


drug use 


0.03 


Weisman et al. (B) (2001) 


MASP 2001-2002 


drug use 


0.2 


Weisman et al. (C) (2001) 


MASP 2001-2002 


drug use 


-0.08 


Weisman et al. (D) (2001) 


MASP 2001-2002 


drug use 


0.37 


Weisman et al. (E) (2001) 


MASP 2001-2002 


drug use 


0.68 


Weisman et al. (F) (2001) 


MASP 2001-2002 


drug use 


0.82 


Weisman et al. (1) (2001) 


MASP 2001-2002 


drug use 


-0.16 


Weisman et al. (J) (2001) 


MASP 2001-2002 


drug use 


0.09 


Weisman et al. (K) (2001) 


MASP 2001-2002 


drug use 


-0.1 


Weisman et al. (L) (2001) 


MASP 2001-2002 


drug use 


-0.1 


Weisman et al. (M) (2001) 


MASP 2001-2002 


drug use 


-0.14 


Weisman et al. (N) (2001) 


MASP 2001-2002 


drug use 


-0.52 


Weisman et al. (2003) 


MASP 2002-2003 


drug use 


-0.01 


Zief 




drug use 


-0.06 


* Effects contained in parentheses are at follow-up. 
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Table C6: Effect Size by Outcome Category: Achievement Tests 






Author 


Program Name 


Outcome Name 


Effect Size* 


Baker & Witt 




achievement (TAAS) 


0.3 


Bergin 


Hilltop Emergent Literacy Project 


achievement tests 


0.38 


Bissell et al. 


YS-CARE 


academic achievement 


0.05 


Chase (A) 




Woodcock Johnson test 


(-0.17) 


Chase (B) 


Hmong-American Partnership Program 


Woodcock Johnson test 


0.37 


Chase (C) 


Hmong-American Partnership Program 


Woodcock Johnson test 


0.46 


Fabiano 


Citizen Schools 


math and English achievement 


0.19 


Foley & Eddins 


Virtual Y 


achievement- reading & math 


0.07 


Huang et al. (2004) 


LA’s BEST 


academic achievement 


(-0.26) 


James-Burdumy 


National Evaluation of 21st Century Learning Centers 


academic achievement 


0.01 


Lauver 




achievement tests 


0.01 


Maxfield et al. (A) 


Quantum Opportunity Demonstration Project: 
Well implemented sites 


academic achievement 


0 


Maxfield et al. (B) 


Quantum Opportunity Demonstration Project: 
Sites with Implementation problems 


academic achievement 


0.13 


Monsaas (A) 


Project Emerge: fourth-grade cohort 


achievement tests 


1.35 


Monsaas (B) 


Project Emerge: fifth- and sixth-grade cohort 


achievement tests 


0.76 


Philliber et al. 


Children’s Aid Society Carerra Program 


academic achievement 


0.15 


Prenovost (A) 




achievement: reading and math 


0.01 


Prenovost (B) 




achievement 


0 


Prenovost (C) 




achievement 


0 


Prenovost (D) 




achievement 


0 


Ross 


ADEPT Drug & Alcohol Community Prevention Program 


achievement total 


0.19 


Tucker & Herman 


Model Program 


academic achievement 


0.48 


* Effects contained in parentheses are at follow-up. 
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Table C7: Effect Size by Outcome Category: Grades 






Author 


Program Name 


Outcome Name 


Effect Size* 


Astroth & Haynes 


4-H Clubs 


grades 


0.29 


Baker & Witt 




grades 


0.3 


Bergin 


Hilltop Emergent Literacy Project 


report cards GPA 


0.66 


Brooks 


LA’s BEST Final Evaluation Report 


grades 


0.42 


Dynarski et al. 


National Evaluation of 21st Century Learning Centers 


grades 


0.04 


Fabiano 


Citizen Schools 


math and English grades 


0 


Gotfredson, Soule SiCross 




GPA 


-0.03 


James-Burdumy 


National Evaluation of 21st Century Learning Centers 


grades 


-0.04 


Lauver 




GPA 


0 


Maxfield et al. (A) 


Quantum Opportunity Demonstration Project: Well-implemented sites 


GPA 


0 


Maxfield et al. (B) 


Quantum Opportunity Demonstration Project: 
Sites with implementation problems 


GPA 


0 


McClanahan 


Summer Career Exploration Program 


grades 


(0) 


Monsaas 


Project Emerge: fourth-grade cohort 


grades 


0.93 


Monsaas 


Project Emerge: fifth- and sixth-grade cohort 


grades 


0.96 


Morrison et al. 




math grades 


0.04 


Neufeld et al. 




grades 


0 


St. Pierre et al. (1992) 


smart Moves Program in Boys and Girls Club 


spelling grades 


0.44 


Weisman et al. (B) (2001) 


Maryland After School Grant Program (MASP): 
2001-2002 school year 


school grades 


-0.28 


Weisman et al. (D) (2001) 


MASP 2001-2002 


school grades 


0.18 


Weisman et al. (F) (2001) 


MASP 2001-2002 


school grades 


-0.23 


Weisman et al. (G) (2001) 


MASP 2001-2002 


school grades 


0.08 


Weisman et al. (H) (2001) 


MASP 2001-2002 


school grades 


-0.01 


Weisman et al. (1) (2001) 


MASP 2001-2002 


school grades 


0.11 


Weisman et al. (J) (2001) 


MASP 2001-2002 


school grades 


-0.02 


Weisman et al. (2003) 


MASP 2002-2003 


GPA 


-0.04 


Zief 




school grades 


0.03 


* Effects contained in parentheses are at follow-up. 
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Table C8: Effect Size by Outcome Category: School Attendance 






Author 


Program Name 


Outcome Name 


Effect Size* 


Astroth & Haynes 


4-H Clubs 


school attendance 


0.25 


Baker & Witt 




school attendance 


0 


Bissell et al. 


YS-CARE 


school attendance 


-0.01 


Dynarski et al. 


National Evaluation of 21st Century Learning Centers 


school attendance 


0.06 


Fabiano 


Citizen Schools 


school attendance 


0.13 


Foley & Eddins 


Virtual Y 


school attendance 


0.11 


Gotfredson, Soule SiCross 




school attendance 


0 


James-Burdumy 


National Evaluation of 21st Century Learning Centers 


school attendance 


0.03 


LaFrance Associates 


Bayview Safe Haven 


school attendance 


-0.01 


Lauver 




school attendance 


-0.07 


LoSciuto & Hilbert 


Woodrock Development Program 


school attendance 


0.26 


Monsaas (A) 


Project Emerge: fourth-grade cohort 


school attendance 


0.52 


Monsaas (B) 


Project Emerge: fifth- and sixth-grade cohort 


school attendance 


0.07 


Oyserman et al. 


School-to-Jobs 


attendance 


0.45 


Prenovost (A) 




attendance 


-0.15 


Prenovost (B) 




attendance 


0.13 


Prenovost (C) 




school attendance 


0 


Prenovost (D) 




school attendance 


0.17 


Tucker & Herman 


Model Program 


school attendance 


0.03 


Weisman et al. (2003) 


MASP 2002-2003 


school attendance 


0.03 


Zief 




school attendance 


0.23 


* Effects contained in parentheses are at follow-up. 
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